Raycast Selection ...

Raycast selection works as expected,  but how can I use it to remove from a current selection?

With the mouse button, include the keys:-

"Ctrl"    =    Exclude selection
"Shift"   =    Toggle Selection (inverse)
"Ctrl+Shift" = Include Selection

That works fine for default selection.  When I use Raycast method...using the Q hotkey ...raycast does not respect  the modifiers

It can be a bit tricky when using "Q", as (for example) by default "Ctrl+Q" = Symmetry Remove-Negative. So when using the Q key+mouse combinations for selections, you need to press in correct order for them to work.

As example:-

Q + MMB + Ctrl Then drag
Q + MMB + Shift Then drag

Either that, or a need to change the hotkeys from other functions that use the combinations..

Ah yes...it was a hotkey conflict with the symmetry hotkey.  Removing the symmetry hotkey Ctrl-Q fixes the issue.

Ah cool....i just discovered I dont even need to have the Q key pressed tool  for raycast selection.

Lmb:  Marquee select

Mmb:  Drag in empty viewport:  Marqee Select with Raycast

      Drag over geometry: Paint Select