From WikiVS, the open comparison website

NX and VNC are remote desktop control protocols widely available on open source operating systems.

NX allows users to create new X11 sessions and forwards all X communication to the remote client.

VNC simply forwards what a logged in user sees on the desktop similar to how video would be transmitted.

[edit] Platforms

Because NX is based on X, it only works on servers where X is present, namely, *nix operating systems (including Mac OSX with X11.app). NX does not run as a server on Windows but can run as a client on Windows.

VNC works on all servers because it simply images the desktop and sends it as pictures instead of X11 calls.

[edit] Performance

NX performs faster than VNC because it works at a higher level, requiring less data to be transmitted. VNC communicates through compressed bitmap screenshot updates of the desktop while NX communicates through compressed X11 calls.

[edit] Multiple clients

Because NX logs into the server as a session, multiple clients are not supported. VNC support multiple clients and is very useful for training purposes.