For computational spectrometers, we propose an inverse-design approach in which the scattering media are topology-optimized to achieve better performance in inference, without the need of a training set of spectra and a distribution of detector noise. Our approach also allows the selection of the inference algorithm to be decoupled from that of the scatterer. For smooth spectra, we additionally devise a regularized reconstruction algorithm based on Chebyshev interpolation, which yields higher accuracy compared with conventional methods in which the spectra are sampled at equally spaced frequencies/wavelengths with equal weights. Our approaches are numerically demonstrated via inverse design of integrated computational spectrometers and reconstruction of example spectra. The inverse-designed spectrometers exhibit significantly better performance in the presence of noise than their counterparts with random scatterers.