You would also need the torrent's hash (which enable you to specify to the tracker which files you want exactly). That's how magnet link [1] works - they contain the torrent's hash and optionally a list of trackers (most clients today support DHT [2], a decentralized tracker-less way to find peers).

[1] https://en.wikipedia.org/wiki/Magnet_URI_scheme

[2] https://en.wikipedia.org/wiki/Distributed_hash_table