okay you got “whom” right the first time but wrong the second time?? why??
also you should never end a clause with a preposition. So these should say “It’s not about with whom you spend the most time, it’s about with whom you had the best memories.”