Yes, chlorine is fairly unstable and will gradually evaporate from water. However, many water treatment plants have begun using chloramine instead of chlorine because it is a more stable disinfectant formed by combining ammonia and chlorine. Chloramine cannot be easily removed from water via evaporation and must be neutralized using a dechlorinator. If you are sure your tap water contains chlorine and not chloramine, you can let the water sit for 1-5 days to allow all the chlorine to evaporate. To speed up the evaporation process, aerate the water with an air stone for 12-24 hours or boil the water for 15-20 minutes. Use multi-test strips to measure the water and make sure no chlorine is left.
For more information, read the full article on dechlorinators.