Files and streams in C++: reading lines, binary data and safe writes
Read from std::istream so tests can use strings, process input line by line, build a console loop, write binary records, detect torn writes and replace files atomically.
Most projects read text line by line; a few have to store data so that it survives a crash. Both start with the same habit: don't tie your logic to a file name.
Read from std::istream, not a file name
Summary summarise(std::istream& in); // the logic
std::ifstream file("readings.csv"); // production
auto s = summarise(file);
std::istringstream text("a,1\nb,2\n"); // tests: no files on disk
auto t = summarise(text);The same goes for output: write to a std::ostream&, and tests pass a std::ostringstream and
compare its .str().
Reading line by line
std::string line;
int line_number = 0;
while (std::getline(in, line)) {
++line_number;
if (!line.empty() && line.back() == '\r') line.pop_back(); // files saved on Windows
if (line.empty()) continue;
// parse(line) ...
}Keep the line number: error messages like "line 14: missing amount" are what a ticket means by "report skipped lines". Process each line as it arrives instead of loading the whole file, and a 2 GB log needs no more memory than a 2 KB one.
A line-based console loop
A small command console is the same loop with a dispatcher:
void run(std::istream& in, std::ostream& out, LinkStore& store) {
std::string line;
while (out << "> " && std::getline(in, line)) {
std::istringstream words(line);
std::string command;
words >> command;
if (command == "quit") break;
else if (command == "add") { /* read the argument, call store, print result */ }
else if (!command.empty()) out << "unknown command: " << command << '\n';
}
}Two rules make it safe for real users: an unknown or malformed command prints a message and the
loop continues (never crash, never throw out of run), and tests drive it with an
istringstream of commands and check the ostringstream. See
the link shortener for Hopwire.
Binary files
For data that may contain any byte (including newlines), write a length before each value:
void write_u32(std::ostream& out, std::uint32_t v) {
unsigned char bytes[4] = {
static_cast<unsigned char>(v), static_cast<unsigned char>(v >> 8),
static_cast<unsigned char>(v >> 16), static_cast<unsigned char>(v >> 24)};
out.write(reinterpret_cast<const char*>(bytes), 4);
}
// record: [key length][value length][key bytes][value bytes]Writing the bytes one by one in a fixed order (here little-endian) makes the file the same on
every machine. Open files with std::ios::binary, and append with std::ios::app.
Torn writes and checksums
If the program or the machine dies mid-write, the last record may be only half there. On startup, read records until one is incomplete, and treat everything from there on as never written:
- Not enough bytes for the header or the declared length? The record is torn: stop there and truncate the file to the last good position.
- A checksum (such as CRC-32) stored with each record catches the other failure: bytes that are all there but wrong. Compute it over the header and the data; on read, recompute and compare.
Flushing (out.flush()) hands data to the operating system; only an OS call such as fsync
forces it onto the disk. Projects usually document which guarantee they give.
Replacing a file atomically
Never rewrite a file in place: a crash halfway leaves neither the old nor the new version. Write a complete new file next to it, then rename it over the old one:
namespace fs = std::filesystem;
void replace_file(const fs::path& target, const std::string& contents) {
const fs::path tmp = target.string() + ".tmp";
{
std::ofstream out(tmp, std::ios::binary | std::ios::trunc);
out << contents;
out.flush();
if (!out) throw std::runtime_error("write failed: " + tmp.string());
} // closed here
fs::rename(tmp, target); // replaces target in one step on POSIX systems
}A reader either sees the old file or the new one, never a mix. This is how the key-value store for Vaultline compacts its log.
Questions and answers
- How do I unit test code that reads a file?
- Make the function take a std::istream& instead of a file name. Production code passes a std::ifstream; tests pass a std::istringstream built from a string literal. No temporary files needed.
- Why open files in binary mode?
- On Windows, text mode turns \n into \r\n when writing and back when reading, which corrupts byte counts, lengths and checksums. Binary mode reads and writes exactly the bytes you give it.
Practise it in a project
These projects put this guide to work, with tickets, tests and code review.
Vaultline
A durable key-value store for Vaultline
Build Vaultline's embedded key-value store: an append-only log that survives crashes, checksums that catch corruption, and compaction that rewrites the log atomically.
- append-only logs
- crash recovery
- checksums
- atomic file replacement
Tracehound
Log analysis for Tracehound
Parse structured log lines, quoted values and all, then compute error rates and p95 latencies so on-call engineers find the slow endpoint in seconds.
- tokenising
- key-value parsing
- percentiles
- aggregation
Hopwire
Link shortener for Hopwire
Build the core of Hopwire's link shortener: turn long campaign URLs into short codes and back again, then give the support team a console to use it.
- hash maps
- number bases
- input validation
- streams
- unit tests
Market Square
Weekly sales report for Market Square
Parse Market Square's sales export, quotes and all, and turn it into revenue per category without letting one bad row spoil the report.
- state machines
- CSV quoting
- error reporting
- aggregation
Read next
- Invariants and all-or-nothing updates: keeping data consistent
Name the rules your data must always obey, enforce them in one place, validate everything before changing anything, keep history append-only, and test invariants with random operations.