Microsoft has a patent showing off a Xbox Controller with removable objects

The USPTO has published a patented named "Game controller with removable controller accessory" which details a modular game controller for Xbox One that can be customized by replacing its objects like keys.

You can find the patent application here and the application is detailed below:

A user input control device, such as a game controller may be used to provide user input to control an object or a character in a video game or to provide some other form of control. A game controller may include various types of controls that may be configured to be manipulated by a finger to provide different types of user input. Non-limiting examples of such controls may include push buttons, triggers, touch pads, joysticks, paddles, bumpers, and directional pads. The various controls may be manipulated to provide control signals that may be mapped to different operations in a video game.