I. Multimedia: Multimodal AI and Multimedia Generation

Discipline: Artificial Intelligence / Multimodal Machine Learning / Generative AI

1. Multimodal AI inputs and outputs

Importance: ★★★★★｜Difficulty: ★★★☆☆☆☆☆｜Level: Core Required

Core concepts:

• AI models can not only process text, but also process and generate different types of data such as Text, Images, Video, Speech, Music, Code, and more.
• Multiple combinations of inputs and outputs can exist, for example:
  • Text → Text
  • Text → Image
  • Text → Video
  • Text → Music
  • Image → Text
  • Image + Text → Text
• Course Emphasis:** Differences in input forms are usually not as significant as differences in output forms, and generation costs and speeds vary widely between output modalities. **

Sample Course:

Input:
"Help me plan my Halloween costume"

Output:
Text → AI gives ideas for clothing

Another case was to give the AI a description and picture of a haunted house and then ask for a sound or video to be generated. The course further demonstrates the combination between text, pictures, video, voice, and music.

Memory association:

Think of AI as a “universal translator”: what used to be just “text ↔︎ text” can now convert between text, images, sound and video.

Self-testing:

Q: What does “multimodal” in multimodal AI mainly mean?

A: AI is capable of processing or generating many different types of data modalities, such as text, images, video, voice, music, etc.

2. Different generation costs for different modalities

Importance: ★★★★☆｜Difficulty: ★★★☆☆☆☆｜Level: Core Required

The core conclusions given by the course:

• Different data types are generated at different speeds.