Lambda in `defun` Captures the Lexical Environment, But in `let` It Doesn't

Question

My example is simplified:

(defvar wtf 10)
(defun f (wtf)
  (lambda ()
    (cl-incf wtf)))
(setq f (f 20))
(setq g (let ((wtf 30))
          (lambda ()
            (cl-incf wtf))))
(list (funcall f)
      (funcall g)
      wtf)
;; ==> (21 11 11)

I have 2 understandings conflicting with each other:

• (defvar wtf 10) made wtf bound dynamically so the variable g would get a closure which captured nothing.
  However, this cannot explain what (f 20) returned.

• (defun f (wtf) ...) created a lexical environment so (f 20) would return a closure which captured the whole lexical environment.
  However, this cannot explain what value the variable g was set.

Of course, neither of them is correct.
So what is the right understanding?

---

It seems that Emacs Lisp is a little different. I am confused.

Answer 1

(defvar wtf 10) made wtf bound dynamically so the variable g would get a closure which captured nothing.

This is correct.

However, this cannot explain what (f 20) returned.

It can if, under lexical binding, function arguments are lexical regardless.

Did you try byte-compiling your code? You should see the likes of this for your function f:

Warning: Lexical argument shadows the dynamic variable wtf

Note that this means you never need to worry that your choice of function argument names might inadvertently bind some dynamic variable of the same name -- potentially causing other functions, called later in the stack, to obtain an unintended value for the variable -- which is liable to be much more confusing/broken behaviour than we get with things the way they are (with the shadowing confined to the lexical scope of the function with that argument).

Answer 2

A long comment instead of an answer. (SE limits the length of a comment.)

---

Hi phils, this is my comment on your answer.

potentially causing other functions, called later in the stack, to obtain an unintended value for the variable

I wrote an example. Is that what you're trying to say? :

;; -*- lexical-binding: t; -*-
(defvar wtf 0)
(defun global-val-of-wtf ()
  wtf)
(defun f (wtf)
  (global-val-of-wtf))
(f 1)
;; ==> 0
;; As contrast,
;; when under dynamical binding rule,
;; (f 1) ==> 1

If this example is indeed the case you wanted to point out, then I think I can understand the reason why:

under lexical binding, function arguments are lexical regardless.

Such a rule really makes sense. (Is it one of the things that makes Emacs Lisp different from Common Lisp?)

But I couldn't find a sentence about this feature in the manual.
All I found was a slightly related sentence in 12.10.5 Converting to Lexical Binding:

The byte-compiler will also issue a warning if you use a special variable as a function argument.

which implies that funtion arguments are lexically bound.

---

Added for Tobias:

May I add a weird case that shows a flaw in phils answer?

We all agree that this is a good feature. But there is an exception, as Tobias's comment mentioned:

;; -*- lexical-binding: t; -*-
(defvar wtf 0)
(setq ans1
      (funcall (funcall (lambda (wtf)
                          (lambda ()
                            wtf))
                        1))
      )
;; ==> 0
(defun f (wtf)
  (lambda ()
    wtf))
(setq ans2
      (funcall (f 1))
      )
;; ==> 1
(setq ans (list ans1 ans2))
;; ==> (0 1)

Save this lisp code to a file. With Emacs 28.1 we get the following strange behavior:

• When we run eval-buffer on the file buffer ans is set to (0 1).
• When we run emacs-lisp-byte-compile-and-load on the file buffer ans is set to (0 0).

The byte-compiler even issues a warning for f:

Lexical argument shadows the dynamic variable wtf

But the behavior we get for the byte compiled case is that one for dynamical binding.

Answer 3

This answer is an extension of Phils answer.

Your question shows the potential danger of locally binding special-declared variables.

The convention of prefixing symbol names with library names normally protects us from accidentally using special-declared symbols for local bindings that should have lexical scope.

But there are exceptions like displayed-month and displayed-year from calendar.el.

One can ensure lexical binding with the help of uninterned symbols.

The principle is described through a simple documented example:

;; -*- lexical-binding: t -*-
(require 'cl)
;; reset:
(unintern 'wtf)
(unintern 'foo)
;; declare `wtf' as special:
(defvar wtf)
(setq wtf 0)
(defmacro f ()
  (let ((sym (make-symbol "wtf"))) ;; make-symbol' returns an uninterned symbol with namewtf'
    (let ((,sym 1)) ;; We substitute here the uninterned symbolwtf'.
       (lambda ()
     ,sym))))
(setq ans (funcall (f)))
;; => 1

It is clear that the definition of the macro f in the above example is cumbersome.

But a special lexlet macro as defined in the following can relieve us from that work.
First comes the library code defining the lexlet macro and below that the user code resembling the simplified introductory example.

The macro lexlet generates uninterned duplicates of all locally bound symbols, locally binds the uninterned symbols to their given values and replaces the locally bound interned symbols in the body with their uninterned counterparts.

The definition of the macro is kept simple. But there is a small caveat. One cannot use the function definition of the locally bound symbols.

;; -*- lexical-binding: t; -*-
;; Here starts the library code.
(defun lexlet--search (data symbol-alist)
  "Replace symbols according to SYMBOL-ALIST in DATA."
  (cond
   ((symbolp data)
    (or
     (alist-get data symbol-alist)
     data))
   ((vectorp data)
    (apply #'vector (seq-map #'lexlet--search data symbol-alist)))
   ((consp data)
    (cons (lexlet--search (car data) symbol-alist)
          (lexlet--search (cdr data) symbol-alist)))
   (t data)))
;; Test:
;; (lexlet--search '(lambda () wtf) '((wtf . var)))
(defmacro lexlet (bindings &rest body)
  "Like `let' but with lexical binding for all symbols in BINDINGS.
If a symbol has a local binding in BINDINGS its binding during
the execution of BODY is lexical even if the symbol is declared as special.
This macro requires `lexical-binding' set to t."
  (declare (indent 1)
           (debug ((&rest (symbol sexp)) body)))
  (let* ((symbol-alist (mapcar
                        (lambda (binding)
                          (let ((symbol (car binding)))
                            (cons
                             symbol
                             (make-symbol (symbol-name symbol)))))
                        bindings))
         (new-bindings (mapcar
                        (lambda (binding)
                          (let* ((old-symbol (car binding))
                                 (new-symbol (alist-get old-symbol symbol-alist)))
                            (cons new-symbol (cdr binding))))
                        bindings)))
    (append
     (list
      #'let
      new-bindings)
     (lexlet--search body symbol-alist))))
(provide 'lexlet)

With lexlet the user code of the introductory example looks friendlier:

(require 'lexlet)
(defvar wtf)
(setq wtf 0)
(defun f ()
  (lexlet ((wtf 1))
    (lambda ()
      wtf)))
(setq ans (funcall (f)))
;; => 1

It is important that the macro extends the lexical environment created by an outer let.
For an example eval with an environment given as argument lexical does not extend an outer lexical environment and is therefore only of limited use for our purpose.

But, lexlet is okay in that regard:

;; -*- lexical-binding: t -*-
(require 'lexlet)
;; reset:
(unintern 'wtf)
(unintern 'lexbound)
(defvar wtf)
(setq wft 0)
(defun f ()
  (let ((lexbound 1))
    (lexlet ((wtf 2))
      (lambda ()
    (list
     lexbound wtf)))))
(setq ans (funcall (f)))
;; => (1 2)