As I said in the comments, the question is extremely broad: pricing an almost arbitrary payoff function for an arbitrary exercise rule for a general stock price process. There are entire books on these issues but here's a short summary. Ultimately, if you want more concrete answers, you need to narrow down your question.
European Options
From the martingale property, we can find prices from discounted expected payoffs under the risk-neutral measure: $$V_E=e^{-rT}\mathbb{E}^\mathbb{Q}[f(K,S_T)].$$ For example, the price of a call option is $C_E=e^{-rT}\mathbb{E}^\mathbb{Q}[\max\{S_T-K,0\}]$. If we make assumptions about the distribution of $S_T$, we can calculate that expectation (or approximate it by simulations in the worst case). Assuming a log-normal distribution, we recover the formulas from Black-Scholes (1973) for vanilla options.
- If we focus on call options for $f$, we can adjust the general formula above using numeraire changes and decompose the option value into exercise probability (value of digital options). This will hold for all stock price models.
- If we focus on a stock price model (say geometric Brownian motion), we can calculate the Greeks and find that $f$ determines the sign of delta, vega and gamma (monotonic $f$ means positive delta, convex $f$ means positive vega and gamma).
There is a second way to price European options: The spanning formula from Carr and Madan (1998):
$$V_E=e^{-rT}\mathbb{E}^\mathbb{Q}[f(S_T)]=e^{-rT}f(F_0) + \int_0^{F_0} f''(K) \text{Put}(K) \ \text{d} K + \int_{F_0}^{\infty} f''(K) \text{Call}(K) \ \text{d} K.$$
Given a complete set of prices for vanilla options, you can now price pretty much every integrable payoff function. A famous example would be the log-contract from Neuberger (1994): Suppose $\text{d}F_t=\sqrt{v_t}F_t\text{d}W_t$. Then, by Ito's Lemma,
\begin{align}
\mathbb{E}^\mathbb{Q}\left[\ln\left(\frac{F_T}{F_0}\right)\right]=-\frac{1}{2}\mathbb{E}\left[\int_0^Tv_t\text{d}t\right]=-\left(\int_0^{F_0}\frac{\text{Put}(K)}{K^2}\text{d}K+\int_{F_0}^\infty \frac{\text{Call}(K)}{K^2}\text{d}K\right).
\end{align}
You get similar formulas if your payoff is $\sqrt{S}$ or $S^3$. Crucially, these formulas are mostly model-independent.
American Options
American options are much harder because they allow you to exercise at any time. Their value is given by $$V_A=\sup_\tau\mathbb{E}^\mathbb{Q}[e^{-r\tau}f(K,S_\tau)],$$ where you additionally maximise over all possible exercise policies (stopping times) $\tau$. There exist a host of numerical procedures to deal with this problem. Two elegant approaches (depending on your payoff function) include maturity randomisation and early exercise decomposition. For example, an American call option is worth
$$C_A = C_E + qS_0\int_0^Te^{-qt}\mathbb{Q}^1[\{S_t\geq B_t\}]\text{d}t - rK\int_0^T e^{-rt}\mathbb{Q}[\{S_t\geq B_t\}]\text{d}t,$$ where $B_t$ is the optimal exercise boundary (which you can find recusrively). We can caluclate Greeks of this value function.
Perpetual Options
I find perpetual options to be among the most interesting objects in finance. They are an entirely different beast because there's no time dependence (elliptic rather than parabolic in PDE language). There are several approaches to pricing perpetual options. You can take American options and consider the limit $T\to\infty$ or derive the corresponding PDE problem. I think the following is more elegant: An American call is worth $V_A=\sup\limits_\tau \mathbb{E}^\mathbb{Q}[e^{-r\tau}f(S_\tau,K)]$. This problem is equivalent to finding the optimal early exercise boundary $B_t$ (``exercise if $S_t>B_t$''). For perpetual options, $B_t\equiv B$ is constant (under a GBM). The option value is thus
$$V_\infty=\max_B \mathbb{E}^\mathbb{Q}[e^{-r\tau_B}]f(B,K),$$ where $\tau_B$ is the first time $S$ reaches the threshold $B$. I really like the term $\mathbb{E}^\mathbb{Q}[e^{-r\tau_B}]$ which looks like an ``expected discount factor''. Intuitively, the optimal $B^*$ resolves the trade-off between a higher payoff and a higher discount factor. Mathematically, $\mathbb{E}^\mathbb{Q}[e^{-r\tau_B}]$ is the Laplace transform of the density function of $\tau_B$. If we assume a stochastic model for $S$, we can either calculate that expectation in closed-form or approximate it via simulations. The case for call/put options and a geometric Brownian motion has a very well-known solution.
