@PawelK I played with some numbers on this.

> Would converting the videos to ASCII arts help in compression? By how much?

To start with, I assume the default terminal size of 80x24 characters and the sixteen-colour terminal palette, and only printable ASCII characters (no UTF-whatever, just byte values of 20 through 7E inclusive).

Each character can have a background and a foreground colour selected independently by means of escape sequences. For sixteen colours, the ASCII sequence looks like
- escape character, 1B
- left bracket, 5B
- two bytes identifying the foreground colour
- semicolon, 3B
- two or three bytes identifying the background colour
- lowercase m, 6D

Any text retains this style until the next modification. In the worst case, you'll have no consecutive colours and thus be forced to alter the foreground and background colour for each character separately, and your backgrounds will all be "bright" colours requiring a three-byte code. This adds up to ten bytes per character - nine in the escape sequence and one printable. With our standard terminal size of 80x24, that's 19.2kB per frame, which at a framerate of 25fps is 3.84Mbps. That's pretty heavy for such a low-fidelity end product, and certainly not competitive with real video codecs. But you can display it with relatively simple tools, just loading the next 1920 characters every forty milliseconds and stuffing them in the terminal.

> What could be the structure of the compressed stream for max space saved?

However, if you're allowed to compress it, you could knock the bitrate down by a lot. A simple compression algorithm would be to store the colour code as a single byte (16 foreground * 16 background = 256 codes) followed by the printable character, which instantly cuts your bitrate by a factor of five: 768kbps. Your tools just need to be smart enough to reconstitute the ten-character escape sequence from the one-byte code. If you have a lot of similarity frame-to-frame (for example, a screencap of a desktop), you might get some additional benefit by running gzip or some other lossless compression tool. For practical purposes, I'm going to pull a number out of my backside and say you wind up around half a megabit per second for a filmed video. So if that's less than your original bitrate, you save some space in compression - but it's still going to look like ASCII art. If there isn't a technical reason you need ASCII art (for example, you're trying to watch it in a TTY with no X server running) you'll certainly get better results just compressing to half a megabit per second in a real video codec.

> Would chroma per character suffice or would chroma per pixel be required?

Because terminals handle colours on a character-by-character basis, you can't address specific pixels of a character to assign them their own colours. And you don't know the font of the user's terminal in advance, so you won't even know what a character's dimensions will be as rendered or which pixels will be foreground/background. Trying to fiddle with specific pixels, even if the terminal provided a means to do so, would be heinously unportable, so let's just leave it at per-character.