You won’t lose anything. Pilani has ample to offer (only if you could assimilate all of it).
A few might say that you will be missing out on many treats, long conversations, splendid night-outs, fun-filled trips, etc. But, you need not be a member of any dept. to cherish those things. You just need a group of people who share similar interests to cherish them.
PS: Joining dept./club/assoc. usually brings you closer to like-minded people on campus.