It gives you a little bit more control over what happens when someone hits a closed port (do you want to drop or reject a packet?), this affects how different nmap scans interpret the port being open/closed/filtered. Also, it's a failsafe in case you don't configure something correctly and something is listening on more than just the loopack (localhost) network interface that you intended. Validating your config with portscanning is great.

Most importantly, you can rate limit and log.

For example, when I have a ufw default deny rule, I get these in my kern.log, and then I can apply a fail2ban rule if someone hits too many closed ports in a certain period (to detect someone portscanning):

kern.log:

    [  494.229628] [UFW BLOCK] IN=tun0 OUT= MAC= SRC=w.x.y.z DST=w.x.y.z LEN=1064 TOS=0x00 PREC=0x00 TTL=59 ID=62111 DF PROTO=TCP SPT=443 DPT=53202 WINDOW=72 RES=0x00 ACK URGP=0 
    [  498.140717] [UFW BLOCK] IN=tun0 OUT= MAC= SRC=w.x.y.z DST=w.x.y.z LEN=135 TOS=0x00 PREC=0x00 TTL=48 ID=8474 DF PROTO=TCP SPT=443 DPT=50851 WINDOW=16 RES=0x00 ACK PSH URGP=0 
    [  500.091643] [UFW BLOCK] IN=tun0 OUT= MAC= SRC=w.x.y.z DST=w.x.y.z LEN=88 TOS=0x00 PREC=0x00 TTL=246 ID=32348 PROTO=TCP SPT=443 DPT=53209 WINDOW=136 RES=0x00 ACK URGP=0 
    [  507.055805] [UFW BLOCK] IN=tun0 OUT= MAC= SRC=w.x.y.z DST=w.x.y.z LEN=135 TOS=0x00 PREC=0x00 TTL=48 ID=8475 DF PROTO=TCP SPT=443 DPT=50851 WINDOW=16 RES=0x00 ACK PSH URGP=0 
    [  524.907273] [UFW BLOCK] IN=tun0 OUT= MAC= SRC=w.x.y.z DST=w.x.y.z LEN=135 TOS=0x00 PREC=0x00 TTL=48 ID=8476 DF PROTO=TCP SPT=443 DPT=50851 WINDOW=16 RES=0x00 ACK PSH URGP=0 
    [  560.619488] [UFW BLOCK] IN=tun0 OUT= MAC= SRC=w.x.y.z DST=w.x.y.z LEN=135 TOS=0x00 PREC=0x00 TTL=48 ID=8477 DF PROTO=TCP SPT=443 DPT=50851 WINDOW=16 RES=0x00 ACK PSH URGP=0 
    [  595.045967] [UFW BLOCK] IN=tun0 OUT= MAC= SRC=w.x.y.z DST=w.x.y.z LEN=79 TOS=0x00 PREC=0x00 TTL=59 ID=14064 DF PROTO=TCP SPT=443 DPT=50131 WINDOW=73 RES=0x00 ACK PSH URGP=0 
    [  595.045985] [UFW BLOCK] IN=tun0 OUT= MAC= SRC=w.x.y.z DST=w.x.y.z LEN=64 TOS=0x00 PREC=0x00 TTL=59 ID=14065 DF PROTO=TCP SPT=443 DPT=50131 WINDOW=73 RES=0x00 ACK PSH URGP=0

/etc/fail2ban/filter.d/portscan.conf

    [Definition]
    # Option: failregex  
    # Notes: Looks for attempts on ports not open in your firewall. Expects the  
    # iptables logging utility to be used. Add the following to your iptables  
    # config, as the last item before you DROP or REJECT:  
    # -A <chain_name> -j LOG --log-prefix "PORT DENIED: " --log-level 5 --log-ip-options --log-tcp-options --log-tcp-sequence  
    # This will place a notice in /var/log/messages about any attempt on a port that isn't open.  
    failregex = \[UFW BLOCK\] .\* SRC=<HOST>

    ignoreregex =

Then if you want to rate limit something (to help prevent DOS), you can do an iptables rule like this (not sure what the equivalent ufw config is):

    -A INPUT -m limit --limit 1/sec -j LOG --log-prefix "INPUT:REJECT:"

Is there some point to blocking portscans? If nothing is listening on the ports then there is little harm to just letting them scan, right?

What I'm getting at is basically, if I have good rate limiting for expensive API's (which I need anyway), and I configure all my services the right way (which I want to do anyway since some software like docker likes to punch through ufw), and I don't care about somebody portscanning a bunch of ports that aren't open anyway is a firewall still a must in your opinion?

I would say so, anything you can to prevent an attacker from gaining useful information is a good deterrent (just like the comment that mentioned hiding version numbers from being displayed). The more deterrents you have, the more you frustrate an attacker and dissuade them from wanting to invest time in exploiting you. So to figure out what open ports you have, if you're using something to block/limit port scanning, it means you're exponentially increasing the time to figure out what ports are open, and possibly forcing them to rotate IP addresses that get banned.

I also don't like wasting CPU or bandwidth, however minuscule, on illegitimate traffic. Unless they are collecting data for a research project or you've hired someone to pentest you, someone port scanning you almost always indicates malicious intent.

If it shows up as blocked in a portscan the attacker will just try again from a different IP, right? think of it as a 403 vs a 404 in HTTP. A 404 says "there is actually nothing here" where as a "403" says "you didn't get to know if there is anything here". I'd rather just tell them that my open ports are 22, 80 and 443. Have at it, since it's cheaper for both of us to let whoever know immediately, and everyone who wants to know will know.

Also, on the CPU/bandwidth part, I don't think sending traffic to a non-listened port is any more expensive than sending it to a firewall. Both are stopped at the kernel, no?

Is there any point to block traffic to a non-listening port any more than is done by just not listening to the port?

Changing IPs is more expensive and requires more effort than being able to scan all ports from one IP, so yes they can certainly do that if they are determined, but the more expensive you make it for them to do that (resources, time, energy), the better a position you are in. Attackers can try anything, but it's defeatist to just let them instead of creating an obstacle.

You have a point that showing you're hiding something could potentially pique someone's interest, but also showing that you have few defenses and obstacles also makes you more appealing to an attacker since it makes you more worth their time. I don't think HTTP status codes are a good analogy, since they are more informative than a port being open/closed. If a port scan shows a port as closed, you don't have any information about whether a service is actually running there, just that you can't access it on a given IP address. If a scan shows filtered for a port then you can infer that it's there, but only accepts requests from a different IP. Also, running SSH on port 22 without any rate limiting is pretty much asking for brute forcing and makes you appealing as a target, even if they have no chance of succeeding it's still wasting resources. I like putting that on a high numbered port if possible, so someone has to port scan (with limited attempts) to even find it.

Depending on how aggressive and thorough you get with nmap (using the -T 5 option, OS/service version detection, and scripting that can actually run active exploits versus just reconnaissance), you can absolutely use up a good amount of CPU/bandwidth. This is a frequent problem for us at work when we are getting pentested, it can be very noisy and disruptive.

Though that resource usage comes more from the volume of traffic versus how it's rejected. Reduce the volume of ports they can try with one IP or in a given time frame, and reduce the resources you spend defending.

However, my understanding is that a firewall would help prevent nmap from doing complete TCP handshake simulations, depending on what type of nmap scan is being done and how the firewall is blocking traffic. So yes, each individual request is almost nothing if that's happening in kernel, but that could easily add up if someone's scanning all 65535 ports and doing an invasive scan. And I haven't even talked about scanning for UDP services, which take more energy on both sides since you don't have a TCP handshake to help you infer information easily.

I think it would help for you to use nmap on your own infrastructure from the position of an attacker: https://www.tryhackme.com/room/furthernmap