Title: On potential theory of Markov processes with jump kernels decaying at the boundary

Abstract: Motivated by some recent potential theoretic results on subordinate killed L\'evy processes in open subsets of the Euclidean space, we study processes in an open set $D\subset {\mathbb R}^d$ defined via Dirichlet forms with jump kernels of the form $J^D(x,y)=j(|x-y|)\mathcal{B}(x,y)$ and critical killing functions. Here $j(|x-y|)$ is the L\'evy density of an isotropic stable process (or more generally, a pure jump isotropic unimodal L\'evy process) in $\mathbb{R}^d$. The main novelty is that the term $\mathcal{B}(x,y)$ tends to 0 when $x$ or $y$ approach the boundary of $D$. Under some general assumptions on $\mathcal{B}(x,y)$, we construct the corresponding process and prove that non-negative harmonic functions of the process satisfy the Harnack inequality and Carleson's estimate. We give several examples of boundary terms satisfying those assumptions. The examples depend on four parameters, $\beta_1, \beta_2, \beta_3$, $\beta_4$, roughly governing the decay of the boundary term near the boundary of $D$.
In the second part of this paper, we specialise to the case of the half-space $D=\mathbb{R}_+^d=\{x=(\widetilde{x},x_d):\, x_d>0\}$, the $\alpha$-stable kernel $j(|x-y|)=|x-y|^{-d-\alpha}$ and the killing function$\kappa(x)=c x_d^{-\alpha}$, $\alpha\in (0,2)$, where $c$ is a positive constant. Our main result in this part is a boundary Harnack principle which says that, for any $p>(\alpha-1)_+$, there are values of the parameters $\beta_1, \beta_2, \beta_3$, $\beta_4$, and the constant $c$ such that non-negative harmonic functions of the process must decay at the rate $x_d^p$ if they vanish near a portion of the boundary. We further show that there are values of the parameters $\beta_1, \beta_2, \beta_3$, $\beta_4$, for which the boundary Harnack principle fails despite the fact that Carleson's estimate is valid.