aMSN Forums
April 23, 2018, 02:52:50 am *
Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length
News: New forum for aMSN !!
 
   Home   Help Search Login Register  
Pages: 1 [2] 3 4 ... 7
  Print  
Author Topic: mimictools 1.0.1 and mimicdecoder 1.0.1.1  (Read 190902 times)
Rv_Charton
Newbie

Offline Offline

Posts: 14


View Profile
« Reply #15 on: March 16, 2007, 04:27:03 pm »

I tried with bash but exactly the same behavior... and the same output file.
About reconstructing the .dat file, could it be done by using the facts that mimicdecoder exits with an error if the .dat file is wrong, and that the first number (the unix time) is certainly pointless (otherwise the "double 0" trick would not work)? I mean by iterating the second number and test the execution of mimicdecoder.
A problem could be that saying 1s per test it would take 14 years to achieve the program on my 428M file...
However it could be done by estimating where the sessions start, and maybe by selecting in the mimicdecoder's code the part that leads to the error.
Juste an idea, please tell me if I'm rambling...

However I'm not yet good enough about shell scripts to do this (but I know a little, so maybe I could manage that), and I know nothing about python...
Logged
hyriand
Newbie

Offline Offline

Posts: 21


View Profile
« Reply #16 on: March 16, 2007, 06:03:59 pm »

Quote from: "Rv_Charton"
I tried with bash but exactly the same behavior... and the same output file.
About reconstructing the .dat file, could it be done by using the facts that mimicdecoder exits with an error if the .dat file is wrong, and that the first number (the unix time) is certainly pointless (otherwise the "double 0" trick would not work)? I mean by iterating the second number and test the execution of mimicdecoder.
A problem could be that saying 1s per test it would take 14 years to achieve the program on my 428M file...
However it could be done by estimating where the sessions start, and maybe by selecting in the mimicdecoder's code the part that leads to the error.
Juste an idea, please tell me if I'm rambling...

However I'm not yet good enough about shell scripts to do this (but I know a little, so maybe I could manage that), and I know nothing about python...


About the large file problem: You could try making mimic2rgb decode the entire file to a raw file. But, once the raw file becomes larger than 2GB, it'll be a problem (and I have no data or free space to test an alternative, using 64bit files). Try mimic2rgb <camfile> <somefiles.raw> and check how far that gets. Just be sure that you have a lot of free diskspace. I might be able to tweak the Makefile so it'll use 64bit file access for you to test.

About reconstructing the .dat file: There are no indicators in the .cam file on where a session begins or ends. The only indication of a 'correct' starting point is that the mimic data will start with a key frame (a 'full' frame), such keyframes happen every so often in a mimic stream. It'd be doable to create an .dat file that enumerates all the keyframes, but that'll just leave you with a HUGE amount of sessions.

Now that I think of it though, I'm not sure wether keyframes are supposed to happen every x frames. If they do, a 'premature' keyframe might indicate a new session as would a resolution change. I'll have to look into that, but don't have the time for that right now.

Another option might be splitting the .cam file up in files of say 50mb. It'd have to cut it right after such a keyframe, so a little tool to do this might be necessary (I think I'll call it mimicslicer).

I'll get back to you on those last two ideas. Which platform do you use btw? And which filesystem?
Logged
Rv_Charton
Newbie

Offline Offline

Posts: 14


View Profile
« Reply #17 on: March 16, 2007, 06:31:40 pm »

Quote from: "hyriand"

About the large file problem: You could try making mimic2rgb decode the entire file to a raw file. But, once the raw file becomes larger than 2GB, it'll be a problem (and I have no data or free space to test an alternative, using 64bit files). Try mimic2rgb <camfile> <somefiles.raw> and check how far that gets. Just be sure that you have a lot of free diskspace. I might be able to tweak the Makefile so it'll use 64bit file access for you to test.


It goes as far as you said: the raw file is 2GB and then it gives me the error message I gave you above.

Quote from: "hyriand"

About reconstructing the .dat file: There are no indicators in the .cam file on where a session begins or ends. The only indication of a 'correct' starting point is that the mimic data will start with a key frame (a 'full' frame), such keyframes happen every so often in a mimic stream. It'd be doable to create an .dat file that enumerates all the keyframes, but that'll just leave you with a HUGE amount of sessions.

Now that I think of it though, I'm not sure wether keyframes are supposed to happen every x frames. If they do, a 'premature' keyframe might indicate a new session as would a resolution change. I'll have to look into that, but don't have the time for that right now.

Another option might be splitting the .cam file up in files of say 50mb. It'd have to cut it right after such a keyframe, so a little tool to do this might be necessary (I think I'll call it mimicslicer).

I'll get back to you on those last two ideas. Which platform do you use btw? And which filesystem?


Indeed, splitting the .cam file was what I was thinking to do (without knowing how), althought I didn't know about keyframes, which regarding to the problem is quite a good news!
Thanks for your help!
I'm using Ubuntu Edgy Eft on a laptop, 512M of RAM and cpu mobile AMD Athlon(tm) XP 2000+. Filesystem is Ext2/Ext3.
Logged
kakaroto
Administrator
Super Power User
*****
Offline Offline

Posts: 9428


View Profile WWW
« Reply #18 on: March 16, 2007, 08:01:23 pm »

spliting the file needs to be done an exact keyframe position, otherwise the second part won't play ,search the forums, something similar was already discussed but I can't remember if a solution was ever found...
about the mimicsplicer finding the sessions, not sure if it will work, the keyframe can change from one stream to another, or it could be all the same, I don't know, it's not a 'correct' method, although it might work a little... about resolution change, I think it's too rare to consider it as a helping factor...
good luck!

KKRT
Logged

KaKaRoTo
hyriand
Newbie

Offline Offline

Posts: 21


View Profile
« Reply #19 on: March 17, 2007, 09:50:13 am »

Quote from: "kakaroto"
spliting the file needs to be done an exact keyframe position, otherwise the second part won't play ,search the forums, something similar was already discussed but I can't remember if a solution was ever found...
about the mimicsplicer finding the sessions, not sure if it will work, the keyframe can change from one stream to another, or it could be all the same, I don't know, it's not a 'correct' method, although it might work a little... about resolution change, I think it's too rare to consider it as a helping factor...
good luck!

KKRT


I agree, it's a dirty hack.. But if it works it works, right? It'll be easy to investigate. Mimicslicer could also cut at every so often MB (making sure it cuts right before a keyframe).
Logged
hyriand
Newbie

Offline Offline

Posts: 21


View Profile
« Reply #20 on: March 17, 2007, 05:08:27 pm »

Ok, after a quick analysis it turns out that keyframes happen every 45 frames in a mimic stream as sent by msn messenger (probably amsn too) and I wrote a small tool that'll detect parts of the stream where an early keyframe is present which indicates the start of a new session. It'll also trigger a new session when the resolution changes. It's called mimicdatregen and is now part of mimictools (which is mimic2rgb + mimicdatregen). The output format is compatible with aMSN and mimicdecoder's .dat files.

Here's an early source tarball of mimictools 1.0.1 (not tested very thoroughly so consider it pre-release). It contains mimic2rgb and mimicdatregen: mimictools-20070317.tar.gz

The Makefile also contains a line to enable 64-bit file support on linux (you'll have to remove a '#' sign to enable it. The Makefile will tell you where). I consider this experimental though as I've never actually tested it myself.
Logged
Rv_Charton
Newbie

Offline Offline

Posts: 14


View Profile
« Reply #21 on: March 17, 2007, 07:40:49 pm »

Great! Thanks a lot!
So I tested mimicdatregen . It works in the sense it finds several sessions. But the first problem I experienced was that the .dat file I obtained looked like:
Quote
1174149936 01174149936 303266721174149936 306294241174149936 353815601174149936 36337384

I corrected it manually thanks to the repeated pattern, but you may  want to fix that in your code.
After that mimicdecoder works, and I can extract specific parts of the stream, so the concept seems to work just fine (even though the sessions are not always real sessions, but I don't mind).
However with my file, here is the output of mimicdatregen:
Quote

Detected new session: early keyframe.
(...)
Detected new session: early keyframe.
mimic2rgb: Invalid header.

I don't know what means this error. I looked at the code but as I know nothing about python...

Otherwise I tried the extended version of mimic2rgb. With the pipe throught transcode, it stops at the same place. But with mimic2rgb alone the output raw file is 5Go big yet, and it is not finished! Looks like it works.
Now I just hope I have enough disk space  :wink: .
Logged
hyriand
Newbie

Offline Offline

Posts: 21


View Profile
« Reply #22 on: March 17, 2007, 09:54:59 pm »

Quote from: "Rv_Charton"
Great! Thanks a lot!
So I tested mimicdatregen . It works in the sense it finds several sessions. But the first problem I experienced was that the .dat file I obtained looked like:
Quote
1174149936 01174149936 303266721174149936 306294241174149936 353815601174149936 36337384

I corrected it manually thanks to the repeated pattern, but you may  want to fix that in your code.

Oops.. Fixed that. Shouldn't happen if you compile with large file support by the way. That code path seems to work properly.

Quote from: "Rv_Charton"
After that mimicdecoder works, and I can extract specific parts of the stream, so the concept seems to work just fine (even though the sessions are not always real sessions, but I don't mind).
However with my file, here is the output of mimicdatregen:
Quote

Detected new session: early keyframe.
(...)
Detected new session: early keyframe.
mimic2rgb: Invalid header.

I don't know what means this error. I looked at the code but as I know nothing about python...

Like we discussed earlier, the session prediction technique isn't perfect. Nothing can be done about that. That error means that the input stream (the .cam file) is corrupted. Each frame has a small header before it which contains among other things, a magic type identifier (the 'fourcc'). In this case mimicdatregen tells you the header was corrupted (or the data is of an unsupported type) and mimicdatregen can't continue its work. Very little can be done about it, but maybe I can convince it to look for a header further in the file.

Quote from: "Rv_Charton"
Otherwise I tried the extended version of mimic2rgb. With the pipe throught transcode, it stops at the same place. But with mimic2rgb alone the output raw file is 5Go big yet, and it is not finished! Looks like it works.
Now I just hope I have enough disk space  :wink: .

Yeah, it seems your shells don't handle large files properly when they go through a pipe (I have no idea if that's a common problem though).
Logged
hyriand
Newbie

Offline Offline

Posts: 21


View Profile
« Reply #23 on: March 17, 2007, 11:12:08 pm »

Ok, new snapshot: mimictools-20070317-2.tar.gz

Changes:
- fixed mimicdatregen's output when compiled without large file support
- merged shared code between mimic2rgb and mimicdatregen into a single helper header
- search for a valid header if a corrupted header is encountered

edit: fixed url
Logged
Rv_Charton
Newbie

Offline Offline

Posts: 14


View Profile
« Reply #24 on: March 18, 2007, 03:05:09 pm »

Thanks!
So I tried it.

-mimicdatgen:
Here's the kind of things I get:
Quote

mimicdatregen: Invalid header (invalid header length: 0x4576).
mimicdatregen: Invalid header (invalid fourcc: 0xACD297AC).
mimicdatregen: Scanning for continuation point.
mimicdatregen: Found continuation point.

and it goes to the end of the file. So it seems to work.
So after that I launched mimicdecoder, and for the first sessions there's no problem, but for the sessions after the fisrt litigious point, when trying to convert it in raw stream, I get:
Quote

mimic2rgb: Invalid header (invalid header length: 0x72E0).
mimic2rgb: Invalid header (invalid fourcc: 0x570C78B5).
mimic2rgb: Scanning for continuation point.
mimic2rgb: Seek failed while scanning for continuation point.
Traceback (most recent call last):
  File "./mimicdecoder", line 98, in run
    stdin_pipe.write(block)
IOError: [Errno 32] Relais brisé (pipe)

 :? Do I have to worry about it?

However mimic2rgb gave me a nice 13Go file, that I tried to convert into xvid using transcode. But it just stops at always the same point, without any error message...
I've been looking in the net for a while about any tool to read that raw RGB stream, so I could figure out if the stream is really damaged, I didn't find anyone. Do you know one?
Logged
hyriand
Newbie

Offline Offline

Posts: 21


View Profile
« Reply #25 on: March 18, 2007, 06:06:39 pm »

Quote from: "Rv_Charton"
Thanks!
So I tried it.

-mimicdatgen:
Here's the kind of things I get:
Quote

mimicdatregen: Invalid header (invalid header length: 0x4576).
mimicdatregen: Invalid header (invalid fourcc: 0xACD297AC).
mimicdatregen: Scanning for continuation point.
mimicdatregen: Found continuation point.

and it goes to the end of the file. So it seems to work.
So after that I launched mimicdecoder, and for the first sessions there's no problem, but for the sessions after the fisrt litigious point, when trying to convert it in raw stream, I get:
Quote

mimic2rgb: Invalid header (invalid header length: 0x72E0).
mimic2rgb: Invalid header (invalid fourcc: 0x570C78B5).
mimic2rgb: Scanning for continuation point.
mimic2rgb: Seek failed while scanning for continuation point.
Traceback (most recent call last):
  File "./mimicdecoder", line 98, in run
    stdin_pipe.write(block)
IOError: [Errno 32] Relais brisé (pipe)

 :? Do I have to worry about it?

That's probably because of a bug in mimic2datregen. Here's a new version (hopefully this'll finally work): mimictools-20070318.tar.gz. With this version you won't be able to extract the session with the broken frame though, at least not with mimicdecoder. The session before and after it should work however.

Quote from: "Rv_Charton"
However mimic2rgb gave me a nice 13Go file, that I tried to convert into xvid using transcode. But it just stops at always the same point, without any error message...
I've been looking in the net for a while about any tool to read that raw RGB stream, so I could figure out if the stream is really damaged, I didn't find anyone. Do you know one?

It's probably the large file problem again. However, you should be able to split the raw stream into smaller pieces since the frame size is entirely predictable.

Each file will hold 9000 raw frames (9000 * 320 * 240 * 3 ~= 1.9GB:
Code:
$ dd if=webcam.raw of=webcam.raw.1 bs=230400 count=9000
$ dd if=webcam.raw of=webcam.raw.2 bs=230400 count=9000 skip=9000
$ dd if=webcam.raw of=webcam.raw.3 bs=230400 count=9000 skip=18000
etc...

Do that about 7 times and you should end up with 7 files of at most 2GB. You can convert those individually.
Logged
Rv_Charton
Newbie

Offline Offline

Posts: 14


View Profile
« Reply #26 on: March 18, 2007, 06:58:59 pm »

Quote from: "hyriand"

That's probably because of a bug in mimic2datregen. Here's a new version (hopefully this'll finally work): mimictools-20070318.tar.gz. With this version you won't be able to extract the session with the broken frame though, at least not with mimicdecoder. The session before and after it should work however.

It works, great! Happiness! :lol: Thanks! Congratulations!
Now I don't need anymore to split the raw file, and I don't have enough free space however. But thanks for that too.

There is one more trouble, that was not there with the first version of mimic2rgb... Now the video is blue; I can see it but the general colour is blue.
However those tools are really great and I think will be usefull to many people.
Logged
hyriand
Newbie

Offline Offline

Posts: 21


View Profile
« Reply #27 on: March 18, 2007, 09:35:57 pm »

Quote from: "Rv_Charton"
Quote from: "hyriand"

That's probably because of a bug in mimic2datregen. Here's a new version (hopefully this'll finally work): mimictools-20070318.tar.gz. With this version you won't be able to extract the session with the broken frame though, at least not with mimicdecoder. The session before and after it should work however.

It works, great! Happiness! :lol: Thanks! Congratulations!
Now I don't need anymore to split the raw file, and I don't have enough free space however. But thanks for that too.

There is one more trouble, that was not there with the first version of mimic2rgb... Now the video is blue; I can see it but the general colour is blue.
However those tools are really great and I think will be usefull to many people.

Ah, right... That's because the old mimic2rgb outputted bgr (which the xvid plugin for transcode expects) and mimic2rgb defaults to rgb.

Edit mimicdecoder, find the line that reads:
Code:
cmd1 = 'mimic2rgb - -'

and change it to:
Code:
cmd1 = 'mimic2rgb - - bgr'


Another option is using ffmpeg, which expects rgb input. The change required to make mimicdecoder use the ffmpeg plugin is described somewhere else in this topic.
Logged
amsn_fan
Newbie

Offline Offline

Posts: 38


View Profile
« Reply #28 on: March 28, 2007, 11:42:53 pm »

I noticed your GUI only finds cam files in the current month for me, it doesnt find anything for last month i.e. in the "February 2007" directory etc.

Apart from that, works fine! Well done!

Edit: Also noticed someone already told you about this!
Logged
Kreuger
Super Power User
**
Offline Offline

Posts: 204


View Profile WWW
« Reply #29 on: March 31, 2007, 03:09:52 am »

It creates the file for me but it's always only 2.0kb in size and when I open it with vlc, nothing shows. I tried running it through a shell for any extra info and it gives me a similar error to the first guy:

Quote
./mimicdecoder
X Error: BadDevice, invalid or uninitialized input device 169
  Major opcode:  147
  Minor opcode:  3
  Resource id:  0x0
Failed to open device
X Error: BadDevice, invalid or uninitialized input device 169
  Major opcode:  147
  Minor opcode:  3
  Resource id:  0x0
Failed to open device
sh: mimic2rgb: not found
transcode v1.0.2 (C) 2001-2003 Thomas Oestreich, 2003-2004 T. Bitterberg
[transcode] auto-probing source /dev/stdin (failed)
[transcode] V: import format    | unknown  (V=raw|A=null)
[transcode] V: import frame     | 320x240  1.33:1
[transcode] V: bits/pixel       | 5.859
[transcode] V: decoding fps,frc | 4.000,0
[transcode] A: import format    | 0x2000  AC3          [48000,16,2]
[transcode] A: export           | disabled
[transcode] V: encoding fps,frc | 4.000,0
[transcode] A: bytes per frame  | 48000 (48000.000000)
[transcode] A: adjustment       | 0@1000
[transcode] V: IA32/AMD64 accel | sse (sse 3dnowext 3dnow mmxext mmx asm C)
tc_memcpy: using sse for memcpy
[transcode] V: video buffer     | 10 @ 320x240
[import_null.so] v0.2.0 (2002-01-19) (video) null | (audio) null
[import_raw.so] v0.3.2 (2002-11-10) (video) RGB/YUV | (audio) PCM
[export_null.so] v0.1.2 (2001-08-17) (video) null | (audio) null
[export_xvid4.so] v0.0.5 (2003-12-05) (video) XviD 1.0.x series (aka API 4.0) |                                                                            (audio) MPEG/AC3/PCM
[import_raw.so] tcextract -i "/dev/stdin" -d 0 -x rgb | tcextract -a 0 -x rgb -d                                                                            0
[export_xvid4.so] Neither './xvid4.cfg' nor '~/.transcode/xvid4.cfg'
[export_xvid4.so] found. Default settings will be used instead.
tc_memcpy: using sse for memcpy
tc_memcpy: using sse for memcpy
(extract_rgb.c) no file type specified, assuming RAW stream

clean up | frame threads | unload modules | cancel signal | internal threads | d                                                                           one
[transcode] encoded 0 frames (0 dropped, 0 cloned), clip length   0.00 s
Traceback (most recent call last):
  File "./mimicdecoder", line 98, in run
    stdin_pipe.write(block)
IOError: [Errno 32] Broken pipe
Logged
Pages: 1 [2] 3 4 ... 7
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.11 | SMF © 2006-2009, Simple Machines LLC Valid XHTML 1.0! Valid CSS!