Once all the icons are displayed, the Up/Down arrow will disappear. To hide some icons, just drag them to the left side and the arrow will appear. Then you can drop the icon to hide it.