The Cloud-Aerosol Lidar and Infrared Pathfinder Satellite Observations (CALIPSO) is unique in its ability to perform profiling measurements of aerosol and cloud layers globally. Detecting the layer boundaries of aerosols and clouds is a crucial step in CALIPSO data retrieval. The CALIPSO team uses the selective iterated boundary location (SIBYL) algorithm based on threshold arrays to find aerosol and cloud layers at different horizontal resolutions. However, threshold arrays could obstruct the detection of optically tenuous layers at a high resolution and may cause overestimation when averaging signals of layer and clear air at a low resolution. Here, a multiscale algorithm using a series of sliding window sizes without threshold setting is proposed based on a pre-defined probability. The results over land and marine areas show that the multiscale algorithm detected 37.41% and 16.36% more layer area than the SIBYL at 1–80 km resolutions at daytime and 1–5 km resolutions at night time, respectively. This indicates that the multiscale algorithm does not need a threshold array, allowing more tenuous layers to be detected, especially at low signal to noise ratios (SNRs). In contrast, the SIBYL detects 4.40% more layer area than the multiscale algorithm at 1–80 km resolutions at nighttime, mainly caused by the large proportion of layer area detected by SIBYL at 20 and 80 km resolutions. This implies possible noteworthy overestimation by the SIBYL at low resolutions. Additionally, the evaluation using the depolarization ratio of ice clouds shows that the extra detected layers by the multiscale algorithm are reliable. Besides, simulation tests show that the multiscale and SIBYL algorithms achieve a 100% true detection rate when SNR is approximately 2 and 4, respectively. The new multiscale algorithm could upgrade the resolution and accuracy of the layer detection of space lidars and reduce the underestimation of layer optical depth due to missing layers.