File transfer protocol question

[someonepoor]

Hi,

   I am writing my own MSN clone so I am tracing aMSN code.
   
   Could anyone explain how direct-connect file transfer works?
   According to my observation, when someone send you a file, it will first send you an INVITE message.
   My question is, sometimes this message from SB server and sometimes from other endpoint client. (with Conn-type...)
   How did the other endpoint know I can use direct-connect method?

   My guess is this is about when aMSN tell the NS server he is on line, it will send its capabilities to server,
   when other receive my aMSN is online, it will also know I can use direct-connect method to transfer files.

   Is this correct? I just can't find this flow in aMSN code. Any help will be appreciated.

[kakaroto]

Hi, welcome to the forums,
I don't understand why people try to implement the protocol, there are libraries out there that you can use! (and I don't understand why people create yet another clone when there are perfectly good clones that exist).
anyways, you seem completely clueless about what's happening, make sure you press Ctrl-D in amsn, to see the protocol log.
the SB server NEVER sends you any invite or anything.. it only relays the messages.. if you receive an INVITE from the SB, then it means the other end point sent it...
the other side doesn't need to know you support direct connect.. if you support msnp2p, then you must support direct connect (don't forget that the official client always had direct connection support, and the protocol is for their clients).
you first get an INVITE for the file transfer, once you accept it, you get an INVITE for the direct connection, you accept it, negociate, etc.. then you get your direct connection, you don't answer it, then the file transfer goes through the SB.
please use google, I don't have time to write hundreds of pages of explanation for a very complicated protocol when you could just :
1 - google it
2 - read the code of other implementations
3 - do your own reverse engineering if you don't understand something.

[someonepoor]

Hi, thank you for the reply.

I am not trying to implement the whole protocol but only enhance the file transfer part. I survey many libs but they have have common flaw that file transfer is very slow in most cases.aMSN deals this very well. However, i need to use python to implement my program. That's why I am doing code tracing.:-)

You information helps much. I will spend more time analyze network packets. Thanks!

[kakaroto]

papyon is a very good python library for msn protocol (and is being used in the amsn2 project (rewrite of amsn in python))

[someonepoor]

papyon is a very good python library for msn protocol (and is being used in the amsn2 project (rewrite of amsn in python))

I successfully transfer my code to papyon.

There is a question. papyon seems not support file transfer yet.
If no one is working on this part, I will implement a file transfer handler myself.
After that I will contribute the code back.

Does anyone start working on this?

[kakaroto]

Hi, great, I'm glad you moved to papyon! This will be a lot less hassle to you and to others since it's one code everyone shares! It will also be helpful for amsn2 since amsn2 is based on papyon.
Yes, unfortunately, there is no file transfer support at the moment.. however lfrb started working on that already, you can find his code here :
http://git.collabora.co.uk/?p=user/lfrb/papyon/.git;a=summary
You'll have to checkout the 'direct' branch to get the file transfer support code.. however, I reviewed his code once and it was not yet ready to be merged, it has quite a few issues, so if you want to work on this, you should fix them first...
Here is my review of the code as I posted it originally in the #papyon channel on IRC... I wrote it in french because lfrb speaks french, sorry :

Sep 10 14:31:18 <KaKaRoTo-KS>   lfrb, http://git.collabora.co.uk/?p=user/lfrb/papyon/.git;a=commitdiff;h=e9dc6ecdf54b5a6433e1b16c29d74fcbcc03423f
Sep 10 14:31:23 <KaKaRoTo-KS>   pkoi c'est 'global'?
Sep 10 14:31:28 <KaKaRoTo-KS>   pkoi c'est pas dans TLPHeader ?
Sep 10 14:35:07 <KaKaRoTo-KS>   lfrb, je viens de me rendre compte de quelquechose.. je crois que les requetes 'transreq' c'est pas TransferRequest, mais plutot TransportRequest.. non? tu en penses quoi ?
Sep 10 14:43:50 *       luckyluke_ has quit (Remote closed the connection)
Sep 10 14:44:45 <KaKaRoTo-KS>   http://git.collabora.co.uk/?p=user/lfrb/papyon/.git;a=commitdiff;h=4f4f264f91dcf4bcc6a58322440a168817d8fa59
Sep 10 14:45:07 <KaKaRoTo-KS>   why is the direct p2p transport saying it can handle msnp2p switchboard messages? :|
Sep 10 14:45:07 <KaKaRoTo-KS>   +    def _can_handle_message(message, switchboard_client=None):
Sep 10 14:47:03 <KaKaRoTo-KS>   the code in
Sep 10 14:47:03 <KaKaRoTo-KS>   +    def _open_listener(self):
Sep 10 14:47:06 <KaKaRoTo-KS>   doesn't make much sense..
Sep 10 14:47:36 <KaKaRoTo-KS>   why increment port when you're just creating the socket (and fail) since 'port' doesn't have anything to do with the socket creation...
Sep 10 14:47:49 <KaKaRoTo-KS>   also, why close the socket, why not retry the bind on the next port and that's it ?
Sep 10 14:58:17 <KaKaRoTo-KS>   lfrb, ton direct transport ne verifie pas que tu recois bien un 'foo'
Sep 10 14:59:04 <KaKaRoTo-KS>   et le nonce, il verifie pas s'il est correct.. .(s'il est pas correct, sa emet pas de connected, mais ca emettra jamais de failed)
Sep 10 14:59:31 <KaKaRoTo-KS>   le timeout de 5 secondes, doit etre fait dans le transport manager, parceque si tu reussi pas a t'authentifier, il devra switcher au switchboard
Sep 10 14:59:57 <KaKaRoTo-KS>   (ou quelconque autre error.. meme dans l'echange de INVITE/200 OK du transreq...)
Sep 10 15:00:32 <KaKaRoTo-KS>   tu verifie chunk.get_nonce() mets tu lui fait pas .upper() (tu le fais avant, mais tu utilise pas la valeur de retour)
Sep 10 15:03:36 <KaKaRoTo-KS>   et puis tu utilise self.__foo pour dire 'foo sent' et 'foo received'...
Sep 10 15:04:13 <KaKaRoTo-KS>   si tu recois quelquechose avant que tu n'ait eu le temps d'envoyer le 'foo', ca risque de te faire skipper le premier chunk (et une erreur, une fois que tu aura fixer le code pour que ca verifie bien le contenu du foo packet)
Sep 10 15:05:19 <KaKaRoTo-KS>   http://git.collabora.co.uk/?p=user/lfrb/papyon/.git;a=commitdiff;h=7c2d3288ad9290d4eb4f7500f1e4b3bcce3b497e
Sep 10 15:05:47 <KaKaRoTo-KS>   tu mets un timeout de 5 secondes pour upnp... alors que ton transport manager devrait failer et utiliser le switchboard apres 5 secondes..
Sep 10 15:05:55 <KaKaRoTo-KS>   faudrait pas que ca soit plus de 200ms a mon avis
Sep 10 15:06:46 <KaKaRoTo-KS>   il y'a aussi bcps de trucs hardcoded (le initial port a 6891, le 'user-agent' pour le upnp description...)
Sep 10 15:11:04 <KaKaRoTo-KS>   lfrb, http://git.collabora.co.uk/?p=user/lfrb/papyon/.git;a=commitdiff;h=2373e0ab41a494132cfe54ca345a0f3dff805288
Sep 10 15:11:22 <KaKaRoTo-KS>   le build_context devrait rajouter 4 * "\xFF" et non pas 4 * "\x00"
Sep 10 15:11:30 <KaKaRoTo-KS>   a la fin
Sep 10 15:13:34 <KaKaRoTo-KS>   faudrait vraiment revoir ton code, parceque comme tu as pu le remarquer.. y'a bcps trop de trucs incomplets...
Sep 10 15:14:01 <KaKaRoTo-KS>   (tu supporte pas te connecter a local + external en meme temps.. tu devrais pouvoir essayer plusieurs connections simultanees)
Sep 10 15:15:06 <KaKaRoTo-KS>   tu supporte pas non plus la bonne detection de qui devrait jouer le role de serveur...
Sep 10 15:15:34 <KaKaRoTo-KS>   si tu envoie un invite transreq, l'autre user DOIT repondre listening=true
Sep 10 15:15:45 <KaKaRoTo-KS>   alors que c'est faux, s'il reponds false, c'est a toi de jouer le role de serveur
Sep 10 15:16:52 <KaKaRoTo-KS>   de la meme maniere, qd tu recois un invite transreq, tu dois pas supposer devoir etre listening.. ca depend de ta propre connectivite.. tu dois prendre en compte si tu es firewalled ou pas, c'est quel type de NAT dans lequel tu es.. est ce que upnp fonctionne ou pas, etc... et comparer avec les settings de l'autre user pour pouvoir decider lequel des deux a plus de chance d'avoir un successful connection
Sep 10 15:17:01 <KaKaRoTo-KS>   et c'est comme ca que tu determine si tu doit etre listening ou non
Sep 10 15:44:05 *       devfil2 (i=4f2af852@gateway/web/freenode/x-skultqupraxqjglt) has joined #papyon
Sep 10 16:04:26 <KaKaRoTo-KS>   lfrb, ah, tu devrais aussi updater les copyrights...
Sep 10 16:05:02 <KaKaRoTo-KS>   direct p2p file, a un header avec copyright ali sabil, alors que c'est toi qui a ecrit le fichier.. donc enleve le des fichiers que tu as creer toi meme (sip/media/webcam/direct..)
Sep 10 16:05:08 <KaKaRoTo-KS>   et rajoute toi sur les fichiers que tu as modifie
Sep 10 16:05:56 <billiob>       lfrb: c'est le moment de devenir célèbre
Sep 10 16:06:16 <KaKaRoTo-KS>   (et rajoute moi aussi sur webcam/p2p/session parceque j'ai pas pense a le faire qd j'ai modifie ces fichiers la)
Sep 10 16:06:20 <KaKaRoTo-KS>   billiob, yo
Sep 10 16:06:39 <billiob>       KaKaRoTo-KS: salut
Sep 10 16:06:52 <KaKaRoTo-KS>   on dirait que les frenchies ont pris ce channel d'assaut aussi
Sep 10 16:07:32 <KaKaRoTo-KS>   sur 11 personnes, il y'a 9 frenchies, et 2 non-french... :p
Sep 10 16:17:53 *       arantes (n=arantes@41.250.123.92) has joined #papyon
Sep 10 16:20:19 <KaKaRoTo-KS>   12 personnes.. 10 frenchies... :p
Sep 10 16:20:40 <KaKaRoTo-KS>   arantes, safi, fterti? :@

Here is a quick translation :

Sep 10 14:31:18 <KaKaRoTo-KS>   lfrb, http://git.collabora.co.uk/?p=user/lfrb/papyon/.git;a=commitdiff;h=e9dc6ecdf54b5a6433e1b16c29d74fcbcc03423f
Sep 10 14:31:23 <KaKaRoTo-KS>   why is it 'global' ?
Sep 10 14:31:28 <KaKaRoTo-KS>   and why isn't it in TLPHeader ?
Sep 10 14:35:07 <KaKaRoTo-KS>   lfrb, I just realized something.. I think that the request 'transreq' isn't TransferRequest, but instead is TransportRequest.. what do you think ?
Sep 10 14:44:45 <KaKaRoTo-KS>   http://git.collabora.co.uk/?p=user/lfrb/papyon/.git;a=commitdiff;h=4f4f264f91dcf4bcc6a58322440a168817d8fa59
Sep 10 14:45:07 <KaKaRoTo-KS>   why is the direct p2p transport saying it can handle msnp2p switchboard messages? :|
Sep 10 14:45:07 <KaKaRoTo-KS>   +    def _can_handle_message(message, switchboard_client=None):
Sep 10 14:47:03 <KaKaRoTo-KS>   the code in
Sep 10 14:47:03 <KaKaRoTo-KS>   +    def _open_listener(self):
Sep 10 14:47:06 <KaKaRoTo-KS>   doesn't make much sense..
Sep 10 14:47:36 <KaKaRoTo-KS>   why increment port when you're just creating the socket (and fail) since 'port' doesn't have anything to do with the socket creation...
Sep 10 14:47:49 <KaKaRoTo-KS>   also, why close the socket, why not retry the bind on the next port and that's it ?
Sep 10 14:58:17 <KaKaRoTo-KS>   lfrb, your direct transport doesn't verifie that you received 'foo'
Sep 10 14:59:04 <KaKaRoTo-KS>   and it doesn't verify if the nonce is correct.. if it's not connect, it doesn't emit a 'connected' signal but it will never emit a 'failed' signal ever
Sep 10 14:59:31 <KaKaRoTo-KS>   the 5 secondes timeout must be done in the transport manager because if you can't authenticate it should use the switchboard
Sep 10 14:59:57 <KaKaRoTo-KS>   (or whatever other error, even in the exchange of the INVITE/200 OK of transreq...)
Sep 10 15:00:32 <KaKaRoTo-KS>   you verify chunk.get_nonce() but you don't do upper() (you do it before but you ignore the return value)
Sep 10 15:03:36 <KaKaRoTo-KS>   and you use self.__foo to say 'foo sent' and 'foo received'...
Sep 10 15:04:13 <KaKaRoTo-KS>   if you receive something before getting the time to send 'foo', it could make you skip the first chunk (and an error once you fix the code to check the content of the foo packet)
Sep 10 15:05:19 <KaKaRoTo-KS>   http://git.collabora.co.uk/?p=user/lfrb/papyon/.git;a=commitdiff;h=7c2d3288ad9290d4eb4f7500f1e4b3bcce3b497e
Sep 10 15:05:47 <KaKaRoTo-KS>   you put a 5 secondes timeout for upnp... while your transport manager should fail and fallback on using the switchboard after 5 seconds!
Sep 10 15:05:55 <KaKaRoTo-KS>   I think that timeout shouldn't be more than 200ms
Sep 10 15:06:46 <KaKaRoTo-KS>   there are also a lot of hardcoded values (like the initial port set to 6891, the 'user-agent' for the upnp description...)
Sep 10 15:11:04 <KaKaRoTo-KS>   lfrb, http://git.collabora.co.uk/?p=user/lfrb/papyon/.git;a=commitdiff;h=2373e0ab41a494132cfe54ca345a0f3dff805288
Sep 10 15:11:22 <KaKaRoTo-KS>   the build_context should add 4 * "\xFF" not 4 * "\x00"
Sep 10 15:11:30 <KaKaRoTo-KS>   at the end
Sep 10 15:13:34 <KaKaRoTo-KS>   you should review all your code yourself because as you noticed there are a lot of issues with it
Sep 10 15:14:01 <KaKaRoTo-KS>   (you don't support to connect to local + external at the same time.. you should try to connect to multiple sockets simultanuously)
Sep 10 15:15:06 <KaKaRoTo-KS>   you also don't support the correct detection of who should act as a server
Sep 10 15:15:34 <KaKaRoTo-KS>   if you send a transreq invite, the other user MUST reply with listening=true
Sep 10 15:15:45 <KaKaRoTo-KS>   which is not true, if it answers false, you should act as a server instead
Sep 10 15:16:52 <KaKaRoTo-KS>   similarly, if you receive a transreq, you shouldn't suppose you'll be listening, it depends on your connectivity, you should take into account whether you are firewalled or not, which type of NAT you are behind, whether upnp worked or not, and compare your settings with the ones from the other user in order to decide which one of you has the most change in having a successful connection
Sep 10 15:17:01 <KaKaRoTo-KS>   that's how you determine if you should be listening or not
Sep 10 16:04:26 <KaKaRoTo-KS>   lfrb, ah, you should also update the copyrights...
Sep 10 16:05:02 <KaKaRoTo-KS>   direct p2p file, has a copyright header by ali sabil, but you're the one who wrote that file.. so remove him from the files you wrote (sip/media/webcam/direct..)
Sep 10 16:05:08 <KaKaRoTo-KS>   and add yourself to the files you modified
Sep 10 16:06:16 <KaKaRoTo-KS>   (and add me too to the webcam/p2p/session because I didn't think of doing it when i modified those files)

ok.. that's a pretty rough and quick translation but it should give you enough comments to get you started and so you know what is missing in that code and in case you test it and it fails, why it failed... I hope it's useful to you!
I hope you can get this fixed and I'll be happy to review your code once you're done fixing lfrb's branch!
Good luck!

[someonepoor]

Hi,

  I download the lfrb branch ver. on Tue, 8 Sep 2009, is this the newest one?
  I am not familiar with git I just download the snapshot of  the latest commit.

  After trace the code. there is a question.
 
  For MSNP2P protocol, the client use SwitchboardP2PTransport to handle the incoming package.
  But how can I write a event handler to handl file event? Because we have only one SwitchboardP2PTransport ,
  it hard to tell the transfer data is for file, webcam or voice.
 
  First I want to modify  the webcam code to handle file transfer.
  After I read the code, I believe this code is not yet complete, even the process flow.

  Any hint that can help me to go on? Thanks!

[kakaroto]

Hi,
no, you're wrong, the version from september 8 is the 'master' branch, it only contains 'stable' stuff, you need the 'direct' branch (at the bottom of the page, you see a list of the branches available). Here's the direct link : http://git.collabora.co.uk/?p=user/lfrb/papyon/.git;a=shortlog;h=refs/heads/direct
You can download the snapshot for the latest commit.
The code you had didn't contain anything relating to file transfers, that's why you may have had a problem when reading the code..
If you look at the p2p.py file, you'll see there's a FileTransferManager that has a _can_handle_message and it checks the EUF_GUID of the invitation to see if it's a file transfer, so it can tell the p2p session manager that it can handle the FT requests...
You should maybe check the branch directly on the first link I gave you, and start reading commits that were added to that branch (from the one tagged 'master' on september 8, to the latest one), just click 'commitdiff' on each commit to read them one by one... This way you'll see exactly how the FT code was implemented and how it evolved and how it hook itself... It will allow you to know how it works and since you'll read one small commit at a time, each having an explanation of what that commit does, it will help you better understand each part of the code.
After you're done, read my comment above about what should be fixed, then get started!

[someonepoor]

Hi,
no, you're wrong, the version from september 8 is the 'master' branch, it only contains 'stable' stuff, you need the 'direct' branch (at the bottom of the page, you see a list of the branches available). Here's the direct link : http://git.collabora.co.uk/?p=user/lfrb/papyon/.git;a=shortlog;h=refs/heads/direct
You can download the snapshot for the latest commit.
The code you had didn't contain anything relating to file transfers, that's why you may have had a problem when reading the code..
If you look at the p2p.py file, you'll see there's a FileTransferManager that has a _can_handle_message and it checks the EUF_GUID of the invitation to see if it's a file transfer, so it can tell the p2p session manager that it can handle the FT requests...
You should maybe check the branch directly on the first link I gave you, and start reading commits that were added to that branch (from the one tagged 'master' on september 8, to the latest one), just click 'commitdiff' on each commit to read them one by one... This way you'll see exactly how the FT code was implemented and how it evolved and how it hook itself... It will allow you to know how it works and since you'll read one small commit at a time, each having an explanation of what that commit does, it will help you better understand each part of the code.
After you're done, read my comment above about what should be fixed, then get started!

OK, I got the code. Its implementation is almost the same as what I have done.
I will check the code and find what I should do.

Thanks!

[someonepoor]

I have complete the interface so the top client received the file complete event. The client now can receive file through switchboard.
Some questions remain:

1. What is the relationship between session manager and session? Seems one p2p session corresponds to one file transfer. Is this correct?
2. I notice that the code import  gupnp.igd , but I don't find any python implementation of  gupnp.igd. Does this code work?

I am implement a direct-file transfer first, because this is part of my project requirement. After that, I will complete the send file part.

[kakaroto]

Hi again!
I'm glad to hear there's some progress!
Yes, a session represents one p2p session (one file transfer, or one webcam session, or one display picture transfer, etc...). The session manager takes care of creating the sessions on new invitations and takes care of sending the message from the switchboard to the correct session (since you can have multiple sessions on a single switchboard).
about gupnp.igd, did you look at the gupnp-igd package ? if it doesn't have python bindings, then maybe it was just some test code, or maybe lfrb wrote the bindings himself but never finished/committed them.
about direct transfer, the code I gave you should support direct transfer.. the name of the branch is 'direct' because it implements direct connection transfers... the whole foo/nonce stuff is the part of direct transfer code.

[someonepoor]

Hi again!
I'm glad to hear there's some progress!
Yes, a session represents one p2p session (one file transfer, or one webcam session, or one display picture transfer, etc...). The session manager takes care of creating the sessions on new invitations and takes care of sending the message from the switchboard to the correct session (since you can have multiple sessions on a single switchboard).
about gupnp.igd, did you look at the gupnp-igd package ? if it doesn't have python bindings, then maybe it was just some test code, or maybe lfrb wrote the bindings himself but never finished/committed them.
about direct transfer, the code I gave you should support direct transfer.. the name of the branch is 'direct' because it implements direct connection transfers... the whole foo/nonce stuff is the part of direct transfer code.

Hi,

Thanks for the explanation.

For gupnp.igd, I already install the libgupnp, but it does not work. But the code seems not just only for testing.
Maybe I will realize what these codes do after reading more code.

[kakaroto]

You're welcome..
maybe you needed python-dev so that installing libgupnp-igd would also compile the python bindings... also note that libgupnp-igd IS NOT libgupnp... it depends on it, but it's not the same package at all...

[someonepoor]

You're welcome..
maybe you needed python-dev so that installing libgupnp-igd would also compile the python bindings... also note that libgupnp-igd IS NOT libgupnp... it depends on it, but it's not the same package at all...

OK, I will try this later.

Another quick question: How can I send a message to the file sender?

Most of the client sample is event-based.  The file transfer event I implement is separated from  conversation event.
Surely I can get the sender mail address by parsing the MSNP2P message, but it is too ugly.

Is there any formal way to send a message to the specific contact?

[staz]

Hi,
I'm also starting to work on this, someonepoor, would you mind sharing your code so we can help each other progress?

Concerning the python bindings, they are in a separate branch of libgupnp-idg

• , but Tester said he is going to merge them soon.

Regards,
Olivier

• http://git.collabora.co.uk/?p=user/lfrb/gupnp-igd/.git;a=summary