The Directory menu XFCE4 panel plugin doesn't follow the change of the base directory via its config dialog. Directory can be changed via xconf only.
It is known bug, which is solved by upstream, see:
- https://bugzilla.xfce.org/show_bug.cgi?id=10331
- http://git.xfce.org/xfce/xfce4-panel/commit/?id=361929416c4e9b064085e44ab8a7e2c5d0e315b9
This patch is applied only in git yet, please, apply this patch for current Funtoo version.