I need almost identical query taken from here: Query for count of distinct values in a rolling date range
I wrote my own version and try to execute it in spark sql environment. However, I get the following error: Error in SQL statement: AnalysisException: Correlated column is not allowed in a non-equality predicate:
This is the query below:
SELECT
fo.orderDate
,f.dist_ids
FROM filtered_orders_date fo
LEFT JOIN LATERAL (
SELECT
count(DISTINCT id) as dist_ids
FROM filtered_orders_date
WHERE orderDate BETWEEN fo.orderDate - 2 AND fo.orderDate -- period of 3 days
) AS f ON TRUE
Is there a way to rewrite the query to get the functionality of getting distinct ids in a date range which would not give the error?