RealMedia Analyzer
v. 0.31.00
User's Manual
============================================================================== [1.] Short description [2.] Limitations [3.] The technique behind [3.1] Other useful utilities you may need [4.] Command line options -c = check file integrity -f = try to fix broken file -d = dump some information about the file -i = rebuild index sections -j = join (multiplex) two RealMedia streams -l = strip operations log and RealMedia Analyzer mark -s = split (demultiplex) stream -u = discard changes according to specified undo file -x = extract part of stream [5.] How do I.. [5.1] Automatically fix broken realmedia file? [5.2] Synchronize video and audio in the realmedia file? [5.3] Extract fixed-rate video/audio from the SureStream file? [5.4] Replace soundtrack in the realmedia file? [5.5] Extract or cut out a clip from the realmedia file? [5.6] Make truncated realmedia file seekable? [5.7] Paste several realmedia files into a single long file? [5.8] Behave myself if the file is still broken and nothing helps?! [6.] Compiling the sources [7.] Contacting the author ============================================================================== [1.] Short description RealMedia Analyzer (rma) is a simple command-line driven utility designed to check integrity and optionally fix errors in the RealNetworks' streaming media files. It also can rebuild index sections e.g. in order to make truncated files seekable, split multiplexed stream into separate audio/video streams, join two RealMedia streams into one multiplexed stream and extract or delete stream parts from the single-stream RealMedia files. ============================================================================== [2.] Limitations - rma has limited ability to work with SureStream (variable bitrate) files, some features will only work with the fixed bitrate files. - rma will not work with files containing multiple DATA sections. - rma will refuse to multiplex/demultiplex/cut/paste streams which contain packets larger than 256 KiB in size. - Since recovery method is very heuristic (not to say "barbaric") it is not guaranteed to work with all files. Actually, it is not guaranteed to work at all. Be sure you've made your backup copies. You have been warned. ============================================================================== [3.] The technique behind RealMedia (rm) is a file format developed by RealNetworks to store and distribute streaming video and audio content. Being designed for real-time network broadcasting, these files however are quite often used for media storage and exchange. Unfortunately, the method used to store media streams in rm files is not error-tolerant, nor error-proof. As a result, you may've experienced glitches (unexpected freezes) when trying to playback files damaged during the network upload or download process. As far as I know, RealNetworks do not offer any utilities to check and fix errors of this kind. However, it is possible to fix (or rather workaround) them. The following technique is used by rma to detect and eliminate broken stream parts. Typical rm file contains two interleaved media streams (audio and video). Each stream consists of a bunch of packets. Each packet has a header part, which contains stream number and packet size. Thus the content of a RealMedia file can be represented by the following figure: +--------+ +--------+ +--------+ +--------+ |stream 0| |stream 0| |stream 1| |stream 0| etc.. |packet 0| |packet 1| |packet 0| |packet 2| +--------+ +--------+ +--------+ +--------+ size00 size01 size10 size02 offset00 offset01= offset10= offset02= offset00+ offset01+ offset10+ size00 size01 size10 Offset of each packet can be calculated as (offset of the previous packet) + (size of the previous packet). When one or several packets are broken (usually just replaced by some baudbarf) this chain is destroyed, and the RealPlayer's file parser is no longer able to determine next packet disposition when a broken packet is encountered. To exclude broken packets from the stream, rma tries to find next intact packet in the stream and then adjust size field of the last encountered intact packet header, so it will point to the next intact packet header. In early rma versions packet size was the only criterion used to determine whether the packet is broken or not. Since v. 0.20.00 rma also optionally checks for correct packet timestamps. Still this isn't something one could call a reliable method. But it works, at least sometimes. [3.1] Other useful utilities you may need============================================================================== [4.] Command line options WARNING: *always* create backup copy of your file when running rma with the -f command. It won't hurt to make a backup copy when running with any other command though. ------------------------------------------------------------------------------ Note for Linux/Unix users: some of the rma commands require file names supplied as their arguments. The file names are enclosed in the slash characters ("/"). However, since in Unix-like environments slash is used as a directory delimiter in the file path, backslash ("\") should be used instead when running Unix version of rma. This only applies to the filenames; all other commands whose arguments contain slashes and backslashes should be left unchanged. Moreover, since most of the Unix shells will treat backslash as a special escape character, you may have to repeat it twice for the shell to interpret it correctly. This applies to all of the commands with backslashes in their arguments. Examples: Windows version: rma.exe -j/c:\tmp\audio.rm//c:\tmp\video.rm/ new.rm Linux version: rma -j\\~/tmp/audio.rm\\\\~/tmp/video.rm\\ new.rm ------------------------------------------------------------------------------ rma command line syntax is rma [command[arguments]] <filename> Command must start with a hyphen. All commands and their arguments are case-sensitive. Available commands are: -c[arguments] = check file integrity (default) With this command rma will check packets disposition in a given RealMedia file. When a broken packet is encountered, rma will stop and display broken packet offset (relative to the file start). You may then explore the file manually with a binary editor. This command accepts six optional arguments: d<n> = set maximum allowed timestamp difference to n (0..32767) milliseconds, 0 means don't care, default is 2000 When performing packets timestamp checking, rma compares current packet's timestamp with the previous packet's timestamp. If the difference exceeds specified number of milliseconds, current packet's timestamp is considered invalid and error is reported. Default value is 2 seconds, but sometimes it may be necessary to increase or decrease it. i = ignore number of packets specified in file header While scanning for bad packets, rma will normally stop processing when the number of processed packets reaches that specified in the file header. Sometimes, however, the header may indicate less packets than are actually present in the stream. If rma reports that the file is possibly truncated, but does not reveal any other errors, it may be a sign of such situation. In this case adding 'i' option may help. s<n> = set maximum allowed packet size to n (1..32767) KiB, default is don't care With this option you can set maximum allowed packet size. Packets with larger values in the header will be considered broken and reported. By default any packet whose indicated size fits within the DATA section bounds is treated as a normal packet. t = disable timestamps checking This option will disable checking for correct timestamps when searching for broken packets. Note that invalid timestamps not always mean that the stream is damaged. Sometimes files created by RealProducer contain multiple timestamp overrides, which seems to be an encoder quirk, probably related to b-frames decoding and presentation. If rma tells you about invalid timestamps but the file is processed normally by the player, just ignore these messages. v = lower verbosity level This argument will disable packets counter, so you may redirect rma's output to a log file without littering the latter with unnecessary garbage. z = stop on timestamp errors Since v. 0.24.00 rma does not stop on timestamp errors and continues to seek through the file until illegal packet size is encountered. If you want it to stop at the first encountered invalid timestamp (as the previous versions did), use 'z' option together with -c command. Note: do not confuse 't' and 'z' options. With 't' option applied timestamp errors will not be reported at all. -f[arguments] = try to fix broken file When issued with this command, rma will not only check packets disposition but also will try to adjust size of the certain packets in order to virtually eliminate broken stream parts. This command accepts the following optional arguments: a<n> = exclude n (0..64, but not more than the number used with 'c' argument) packets after first found intact packet, default is 0 If the fixed file still freezes at the same position during playback, increasing this value may help. However it should not exceed the minimum intact packets chain length (see the 'c' option description below). b<n> = exclude n (0..64) packets before last intact packet, default is 0 For the same purpose that the previous parameter. c<n> = use minimal size of n (2..32767) packets chain when searching for unbroken stream parts, default is 4 Set minimum length of the packets chain to search. Default value should suit in most cases, but sometimes it may help to increase it. d<n> = set maximum allowed timestamp difference to n (0..32767) milliseconds, 0 means don't care, default is 2000 When performing packets timestamp checking, rma compares current packet's timestamp with the previous packet's timestamp. If the difference exceeds specified number of milliseconds, current packet's timestamp is considered invalid. Default value is 2 seconds, but sometimes it may be necessary to increase or decrease it. i = ignore number of packets specified in file header With this option, rma will not stop scanning when the number of processed packets reaches that specified in the file header, but will continue until the file end is reached. See description of this option above. m[suppression level] = suppress logging Since version 0.25.00 rma puts its signature at the end of processed RealMedia file, so it becomes possible to determine later whether the file has been altered by rma or not. Since v. 0.30.00 even more advanced logging system is implemented. When run with a command that may possibly alter the rm file contents, rma logs all performed operations and appends this information to the end of processed file. Operations log can be displayed with the -d command. There is a specific command to strip this information from the file as well. With 'm' option you can control the amount of information being logged. The 'suppression level' argument may take one of the following numeric values: 1 = do not update log. Will keep the log if it was present, but will not add any new information to it. 2 = mark only. Will strip operations log but leave the rma signature at the end of the file. 3 = suppress completely (default). Will strip both operations log and signature if present, and will not add any new ones. If 'm' option is not specified, full operation logging is performed. n<n> = only fix n (1..32767) broken packets, default is all Use this to ensure that the glitch was eliminated. E.g., use n1 for rma to try to fix only the first error, then try to play back the file. If it is fixed, go on to fixing the next error, otherwise try to play around with 'a' and 'b' parameters. '-fb1' usually does the trick. p<n> = force maximum packet size (intact packet criterion) to n (16..32767) bytes This value is used to recognize intact packets among broken ones when searching for the intact packets chain. Do not confuse it with the size set with 's' option (see below). By default rma takes this value from file header, but if it is less than 1024 it is always set to 1024. s<n> = set maximum allowed packet size (broken packet criterion) to n (1..32767) KiB, default is don't care With this option you can set the maximum allowed packet size. Packets with larger values in the header will be considered broken and eliminated. By default any packet whose indicated size fits within DATA section bounds is treated as a normal packet. You should use 's' option if during the fixing process enormously large new packets' sizes are being displayed. t = disable timestamps checking It is recommended to fix errors in the file (if any) with this option applied, and then when all the other errors are fixed try to fix the ones related to invalid timestamps. This kind of errors is usually hardest to fix, however invalid timestamps not always indicate broken stream and not always result in playback freezes. v = lower verbosity level (see above) z = try to fix timestamp errors Since v. 0.24.00 rma does not try to fix timestamp errors by default, and continues to seek trough the file until illegal packet size is encountered. If you want it to try to fix timestamp errors as well (as the previous versions did), use 'z' option together with -f command. Note: do not confuse 't' and 'z' options. With 't' option applied, timestamp errors will not be considered at all (i.e. will not even be reported). /undo_filename/ = create an undo file with the specified pathname If undo filename is specified, all changes made to packet headers during the autofix process will be logged to the undo file so it is possible to discard them later (see -u command description). Note: undo feature is a "poor man's backup". If you have enough of free disk space, it is better to make a backup copy of the file being processed. -d[argument] = dump some information about the file When issued with this command, rma writes some information retrieved from rm file to the console (stdout). This command accepts one of the following mutually exclusive arguments: b = dump brief information about the file Only the average bitrate, file duration, audio and video bitrate, video frame size and approximate framerate are displayed. g = dump general information Will dump miscellaneous information retrieved from file header and the RealMedia Analyzer operations log if the latter is present. This is the default behavior. l = display operations log only Will display the RealMedia Analyzer operations log if it is present. i = dump file index Will verbosely dump rm file index sections. -i[arguments] = rebuild index sections With this command rma will try to rebuild index for each stream contained in the rm file. Use this command to make truncated file or file with broken index seekable. Note: You always have to use -f command before using -i command if you're not sure if the file has any errors. Available arguments for this command are: a = index all keyframe packets You may encounter a RealMedia file which contains a stream consisting of keyframes only (no diff frames). In this case usual algorithm used to build index will not work (only first frame will be indexed). With the 'a' option you can force indexing of each packet belonging to a keyframe. This will make any RealMedia file seekable, but you may experience some glitches when try to forward/rewind newly indexed file during playback. m[suppression level] = suppress logging (see above) v = lower verbosity level (see above) -j[arguments] = join (multiplex) two RealMedia streams With this command you can multiplex or paste two RealMedia streams. Only single-stream files can be multiplexed or pasted together (but rma will not make any checks for number of streams, it will simply take stream 0 from each file). Normally, when multiplexing two files the first file contains audio stream and the second file contains video stream. When pasting two files they both must contain streams of similar nature. It is usually harmless to paste two audio streams, but trying to paste together two video streams rarely succeeds (it seems that RealPlayer is very sensitive to the changes in packets' timestamps in the newly created stream). Available arguments for this command are: /firstfilename/ = pathname of the file containing the first stream, default is stream00.rm /secondfilename/ = pathname of the file containing the second stream, default is stream01.rm With these two arguments you can specify names of the files containing the first and second stream. Default names are stream00.rm and stream01.rm in the current directory. Which stream will be numbered 0 in the resulting file is determined by order of appearance in the command line. i = build index after join With this option rma will automatically build index in the output file. Otherwise you will have to run rma once more with -i command explicitly. m[suppression level] = suppress logging (see above) p = paste instead of multiplex If this option is present, files will be pasted together. Otherwise they will be multiplexed. s = try to synchronize streams If this option is present, rma will try to equalize streams' durations. This is done by means of adjusting packets' timestamps in the stream with lesser duration. Note: it is dangerous to change timestamps in a video stream. In most RealMedia files audio stream duration is slightly lesser than video stream duration. This is normal. In any case, you should avoid using 's' parameter unless absolutely necessary. Note: 's' parameter is ignored if used simultaneously with 'p' parameter. q[l]<time> = set relative stream 0 to stream 1 skew (delay) in milliseconds (0..10000, signed). ql means set logical skew, otherwise set physical skew With this argument you can set relative video-to-audio start delay. Negative number means the first (zero) stream starts earlier. "Logical skew" means that only appropriate fields in the file header will be adjusted. However, these fields may be ignored by the player. "Physical skew" means that the actual packets' timestamps will be adjusted instead. If requested skew is negative the second stream will be affected, otherwise adjustment will be applied to the first stream. Remember that it is dangerous to change timestamps in a video stream. You should keep this in mind when using the 'q' parameter. Note: 'q' parameter is ignored if used simultaneously with 'p' parameter. v = lower verbosity level (see above). -l = strip operations log and RealMedia Analyzer mark, if any With this command rma will strip operations log and signature from the end of processed RealMedia file, if they are present. -s[arguments] = split (demultiplex) stream This command will demultiplex RealMedia stream to separate video and audio streams. By default stream 0 and stream 1 are extracted and written to the files with names equal to the input file name and extension replaced with the stream number (i. e. '000' and '001'). Available arguments are: i = ignore number of packets specified in file header With this option, rma will not stop processing when the number of extracted packets reaches that specified in the file header, but will continue until the file end is reached. See description of this option above. m[suppression level] = suppress logging (see above) p</directory prefix/> = use specified drive/directory for output files (default is current directory) Specified directory will be used to store extracted streams. By default files are written to the current directory. x<n> = extract n'th stream from the file Will extract specified stream instead of streams 0 and 1. This command is useful for converting SureStream files into conventional fixed-bitrate files. v = lower verbosity level (see above). -u[arguments] = discard changes according to specified undo file With this command you can restore original packet headers from the previously created undo file (see -f command description). This command is useful if autofix was ineffective and you would like to try -f command with a different parameters set. You should specify undo file name instead of RealMedia file name when using -u command. Example: rma -f/test.und/ broken.rm will create undo file named test.und in the current directory; rma -u test.und will restore original broken.rm. The only available argument for this command is: m[suppression level] = suppress logging (see above). -x[arguments] = extract part of the stream With this command you can extract or delete parts of a RealMedia stream. It only works with the single-stream files. If you want to extract or delete part of a multiplexed audio/video, you should demultiplex it first and use '-x' command on each stream, then use '-j' command to re-multiplex them. Again, rma will not check if the file does not contain multiple streams, it merely takes packets from the stream numbered 0. Resulting file (either extracted chunk or the file with a deleted chunk) is written to the file with the name equal to the input file name and extension replaced with 'cut'. Available arguments for this command are: /<start time> (default is 00:00:00.00) Set start time of the stream chunk to extract. Time record format is hh:mm:ss.cs. \<end time> (default is end of stream) Set end time of the stream chunk to extract. d = delete part of stream instead of extract If this option specified, stream chunk will be deleted instead of being extracted. k = ignore keyframes Normally rma will search for keyframes with the timestamps most close to the specified start and end time. With this option, however, rma will consider any packet with the closest timestamp as a valid chunk start/end, regardless of it being a keyframe packet or not. Note: in some files there are only keyframes present (no diff frames). In audio streams all packets are also usually marked as a keyframe packets. To correctly extract chunk from such files you will need to use 'k' option with the -x command. m[suppression level] = suppress logging (see above) o = write old-style file header Since v. 0.30.00 rma uses new (V1) RealMedia file headers in the files it creates. With this option you can enforce old (V0) header to be used. p</directory prefix/> = use specified drive/directory for output file (default is current directory) Specified directory will be used to store extracted chunks. By default files are written to the current directory. t = ignore timestamps If this option specified, rma will not try to correct packets' timestamps in the extracted stream chunk or in the file with deleted stream chunk. Normally rma will adjust extracted chunk's packets' timestamps so that the first packet has timestamp of 0 milliseconds. However, it may cause problems with video synchronization and/or playback. Leaving timestamps not corrected in the extracted chunk usually doesn't hurt. However, when deleting stream chunk, use of 't' parameter will lead to a pause during playback. Pause duration will be equal to the deleted chunk duration. Since it is quite dangerous to change timestamps in the video streams, it is recommended to use this option whenever possible. ============================================================================== [5.] How do I.. This section contains command line examples for the most common tasks you may use rma for, with minimum explanations when necessary. [5.1] Automatically fix the broken realmedia file? rma -f broken.rm If it doesn't help, restore the backed up copy and try rma -fa1b2 broken.rm If something strange happens, try rma -ft broken.rm [5.2] Synchronize video and audio in the realmedia file? First, split the file into separate video/audio streams: rma -s broken.rm You will get two files: broken.000 and broken.001. Now try to play with video/audio relative time shift. E.g., set audio skew to 500 milliseconds (0.5 sec): rma -jql500/broken.000//broken.001/ fixed.rm If it's getting worse, try to change the skew direction: rma -jql-500/broken.000//broken.001/ fixed.rm If nothing happens, try to force skew: rma -jq500/broken.000//broken.001/ fixed.rm or rma -jq-500/broken.000//broken.001/ fixed.rm If it still doesn't help, here's the last resort: rma -js/broken.000//broken.001/ fixed.rm Don't forget to reindex newly created file: rma -i fixed.rm [5.3] Extract fixed-rate video/audio from the SureStream file? First, dump the stream information into a text file: rma -d surestream.rm > log.txt Browse log.txt with an ASCII text viewer. Find out numbers of streams with MIME type x-pn-realaudio and x-pn-realvideo. They are the fixed rate audio and video stream respectively. Say, they are stream 6 and stream 8. Extract these streams: rma -sx6 surestream.rm rma -sx8 surestream.rm You will get two files: surestream.006 and surestream.008. Now, multiplex them into a single file: rma -j/surestream.006//surestream.008/ fixed.rm Don't forget to reindex newly created file: rma -i fixed.rm Note: some SureStream files do not contain fixed rate streams at all. In this case you're out of luck. [5.4] Replace soundtrack in the realmedia file? Extract video stream from the file. Usually it's stream 1. rma -sx1 broken.rm Encode new soundtrack into realaudio format. You will need RealProducer to accomplish this task. Now multiplex extracted video with the new sound: rma -j/newsound.ra//broken.001/ fixed.rm Don't forget to reindex newly created file: rma -i fixed.rm [5.5] Extract or cut out a clip from the realmedia file? Let's extract a 30-sec clip from the test.rm, starting at 50 sec mark and ending at 1:20. First, demultiplex test.rm: rma -s test.rm You will get test.000 and test.001. Now let's extract chunks from both streams: rma -xtk/00:00:50.00\00:01:20.00 test.000 ren test.cut audio.rm rma -xt/00:00:50.00\00:01:20.00 test.001 ren test.cut video.rm Note, we didn't forget to rename 'test.cut'! Now let's multiplex two clips: rma -ji/audio.rm//video.rm/ clip.rm [5.6] Make truncated realmedia file seekable? rma -i broken.rm If it doesn't work try rma -ia broken.rm [5.7] Paste several realmedia files into one long single file? Let's paste two RealAudio files 1.rm and 2.rm into one file 3.rm: rma -jp/1.rm//2.rm/ 3.rm rma -i 3.rm Note: trying to paste video usually doesn't give good results. [5.8] Behave myself if the file is still broken and nothing helps?! Fall on the floor and cry loudly. Trying to brake nearby furniture also may help. When finished, try this: 1) Locate broken packet: rma -c broken.rm rma will give you last good packet's timestamp. Write it down. Say it's e.g. 00:12:05.32. 2) Run autofix: rma -f broken.rm 3) Demultiplex the file: rma -s broken.rm 4) Delete broken part from the video stream: rma -xdt/00:12:05.30\00:12:05.35 broken.001 5) Multiplex video and audio: rma -ji/broken.000//broken.cut/ fixed.rm ============================================================================== [6.] Compiling the sources Full source code is included with the distribution since v. 0.20.00. rma is written in a portable Pascal dialect and can be compiled with the Free Pascal compiler for a variety of target platforms. Build scripts are provided to compile rma for Windows and Linux. To build rma from the source, do the following:
- A command-line utility called rmedit is (or at least used to be) supplied with RealProducer. You may find trial version (and order a pro version) at the RealNetworks web site. With this program you can edit CONT (content description) section of rm files and also paste several RealMedia files into one file. Probably rmedit's paste feature is more reliable than that of rma, but I'm not sure about it.
- A C library for RealMedia streams parsing can be found at the Moritz Bunkus' web site.
- More open source RealMedia tools can be found at the Helix Community web site.
Compiling for other targets should be straightforward as well, provided you have working Free Pascal installation configured for specified target. It is also possible to compile rma with Delphi, although the correctness of obtained executable was not thoroughly tested. You can either use dcc32 command line compiler, or rename rma.pas to rma.dpr and open it with the Delphi IDE. The DELPHI conditional symbol must be defined, with -DDELPHI switch when using command line compiler, or in the project settings when using the IDE. Other Pascal compilers may work, but this is not guaranteed. If you still want to try, there are a few prerequisites:
- download and install Free Pascal compiler;
- unpack rma sources to the directory of your choice, make it the current directory;
- to build for Windows with Windows-hosted compiler, run buildwin32.cmd script;
- to build for Linux with Linux-hosted compiler, run buildlinux script (you may need to make it executable with appropriate chmod first);
- to cross-build for Linux with Windows-hosted compiler, run buildlinux.cmd script.
============================================================================== [7.] Contacting the author You may contact me by e-mail. Send a message to alexeysp at yahoo com or alexeysp at i ua (insert appropriate symbols in place of 'at' and spaces). Latest version of rma can be downloaded from http://www.i.com.ua/~alexeysp/rma/rma.html or obtained from the author via e-mail. Comments, suggestions, bug reports are welcome. ============================================================================== Initial document version: 31.Oct.2000 Last updated: 31.Mar.2012
- UNIX symbol must be defined for Linux and other Unix-like targets;
- BIG_ENDIAN symbol must be defined when compiling for big-endian target platforms.
- Build date is taken from datetime.inc, so you have to provide the means for dynamic generation of this file containing the actual date and time. It can be done either by utilizing the standard OS commands or by using the datetime utility included along with the rma sources.