Minecraft lag

Just a regular Minecraft player here with server lag issues, and I haven’t been able to find anybody with a similar issue, or if there is somebody, anybody, I guess I don’t know what to search anymore. But the problem is that my friend has Minecraft server, he has it port forwarding, decent amount of ram dedicated to server, and has a very stable connection.

I also have a very stable internet connection, and very good speed, it can get up to 100 mb/s depending on usage in my house, however for some reason, when there are more players on the server, upwards of about 4 other players, I just start lagging like crazy but only me.

My PC is fine, I get a lot of FPS so I’m guessing it’s not hardware issue, but aside from that I have no idea what could it be and how to approach this.