Cipher

All three are symmetric block ciphers using the same AES algorithm, but with different modes of operation that change how blocks are chained together.
AES-GCM (Galois/Counter Mode) is the modern default - it provides both encryption and authentication, meaning any tampering with the ciphertext is detectable. Prefer this unless you have a specific reason not to.
AES-CBC (Cipher Block Chaining) is a classic mode that encrypts each block using the previous ciphertext block. It does not provide authentication on its own.
AES-CTR (Counter Mode) turns AES into a stream cipher. Like GCM it is parallelizable, but like CBC it provides no authentication.

RSA-OAEP is an asymmetric encryption scheme - it uses two different keys: a public key to encrypt and a private key to decrypt. This means you can share the public key openly and anyone can encrypt data that only you can read. The downside is that RSA can only encrypt small amounts of data (roughly 190 bytes for a 2048-bit key, ~446 bytes for 4096-bit). For anything larger, the standard approach is to encrypt the data with AES and encrypt only the AES key with RSA.

For the AES modes, the password is never used directly as a key. Instead, a Key Derivation Function (KDF) stretches it into a 256-bit AES key. A random 16-byte salt is generated on every encryption and embedded in the output, so the same password produces a different ciphertext each time.
KDF - currently only PBKDF2 is available natively in the browser's Web Crypto API. scrypt and Argon2id are stronger alternatives (memory-hard, more resistant to GPU brute-force) but require a third-party library and are shown here as disabled placeholders for future support.
Hash - the internal hash function used by PBKDF2 as its pseudorandom function. SHA-256 is the standard choice. SHA-384 and SHA-512 are slightly stronger and produce larger intermediate values, but the practical security difference is negligible when combined with a high iteration count.
Iterations - how many times PBKDF2 repeats the hash internally. More iterations mean slower key derivation, which makes brute-force attacks proportionally slower. 100,000 is the current OWASP recommendation for PBKDF2-SHA-256.

Normally you encrypt with a raw password - the characters you type are passed directly into PBKDF2, which derives the actual key from them. "Hash pw" adds an extra step before that: your password is first run through the selected hash function (SHA-256, SHA-512, etc.) and the resulting hex string is then used as the input to PBKDF2 instead of the original password.

This is an uncommon setting and "Raw" is the right choice for most people. The option exists for interoperability: if you have data that was encrypted by another system which pre-hashed the password before key derivation, you need to reproduce that same step here to decrypt it successfully. If you are encrypting fresh data yourself and no external system is involved, leave it on Raw.

The ciphertext is always a JSON object. Depending on the "Include metadata" setting it may also contain the algorithm, KDF settings, and iteration count - or just the bare minimum: salt, IV, and encrypted data. The output format controls how that JSON is encoded for transport:
Raw outputs the JSON directly as plain text - human-readable and self-describing.
Base64 and Hex encode the entire JSON string, producing a more compact opaque string. The decryption input accepts all three formats and detects them automatically.

When enabled, the encrypted output contains a self-describing JSON object with all parameters needed for decryption: algo, kdf, hash, iterations, salt, iv, and data - obviously, the password, of course, is not included, you have to remember it.
When disabled, only salt, iv, and data are included. The output is slightly smaller and reveals less about the encryption setup, but the recipient must know which algorithm, KDF, hash, and iteration count were used - otherwise decryption will fail. This tool handles missing fields by falling back to the currently selected values, with a warning.

In most cases: yes, leave it on. The metadata fields (algo, salt, iv, etc.) are not secret - in every standard encryption protocol these values are transmitted alongside the ciphertext, because they are required for decryption and knowing them does not help an attacker break the password. The only thing that must stay secret is the password itself.

Turn it off if you have a specific reason: for example, you are producing output for an external system that expects a fixed compact format, or you want to avoid revealing which tool and settings you used. The latter is a minor obscurity benefit at best and is generally not a substitute for a strong password.

The cryptographic primitives used here (AES-256, RSA-OAEP, PBKDF2) are industry-standard algorithms provided by the browser's built-in Web Crypto API - no third-party libraries are involved. Everything runs locally in your browser; no data is sent anywhere. That said, the security of any encrypted data is only as strong as the password protecting it - use a long, random password and store it safely.