@srijan Sounds like poor interaction between packages. I suspect it's with helm but that's probably just my personal bias against big complicated packages 🤷
@srijan Apologies, I took a quick look at your github, thought I saw helm in your init.
Are you sure you are measuring scrolling? I tried profiling and got similar report as yours, but I realized I didn't spend much time scrolling compared to time in the minibuffer to run M-x profiler-stop. When I scrolled for a longer time, it looked completely different. However I'm not seeing any scrolling slowness. (I'm also using vertico/orderless).
Probably better practice would be to bind a key to profile-stop, but I didn't try it since I'm not seeing the issue you are seeing.