I'd like to use BitTorrent for sharing files with Nessie. So, I started doing some research about private trackers. It took me quite a while to scrape together enough information about BitTorrent protocols to figure out how private trackers work. (It would have helped if I had found the official BitTorrent specifications sooner.) It turns out to be quite simple.
Basically torrent trackers and clients exchange dictionaries of meta-data. In the meta-data sent from the client to the tracker is a passkey (with the key name 'key'). The passkey is per-user and added to the announce URL in the .torrent file which is dynamically generated for each registered user that downloads it. The tracker then uses that key like a session key in a web app. The key can be used for connection limiting, ratio tracking, IP restriction, etc. Useful!
When trackers send data back to the clients, a private flag bit is set (with the key name 'private'). Well behaved clients will then refrain from leaking the passkey/announce URL to other users via DHT, peer exchange, etc. But, if a client misbehaves, and the tracker security catches it, it's trivial to invalidate a passkey and plug the leak.