A boost cancel is when you boost and then immediately push the left thumbstick in the opposite direction to the one you are boosting in. If done quickly enough, it will look like you never boosted.
This is used during the Garage tutorial, as the final task you must do for Gum is a boost. The challenge is only completed once the boost has ended, so a boost cancel is almost 3 seconds faster than doing a full boost. This optimization was discovered by FractalFusion in April 2007, but they did not use a boost cancel to do it.
Technically, a proper Boost Cancel is only when you cancel the boost while your character is still going too slow for a normal skid; otherwise, you're simply skidding to a halt during a boost. There is no functional difference between these two scenarios.