My .bash_history weights 1.7Mo (almost 100k lines). I set a very high HISTSIZE since I don't see any reason to lose this data.

One reason might be that bash reads this whole file into memory on interactive startup and rewrites it completely when shutting down.

I second this, I think it is good practice to rotate the history file manually when it reaches several MBs. (Using zsh, the main symptoms of a large history file is sluggishness when using ^R and a lag of a few tenths of a second when closing a terminal.)

For now I don't feel any annoyance, the shell startup is instantaneous and it closes instantaneously as well. If I'm starting to feel a slow down I might rotate the log, but I guess that's not happening soon.