Boost Filesystem Version 3

Deprecated Names and Other Features
Breaking Changes
Design

Version 3 is a major revision of the Boost Filesystem library. Important changes include:

A single class path handles all aspects of internationalization, replacing the previous template and its path and wpath instantiations. Character types char and wchar_t are supported. This is a major simplification of the path abstraction, particularly for functions that take path arguments.
New class path members include:
- path::has_stem
- path::has_extension
- path::is_absolute. This renames is_complete(), which is now deprecated.
- path::is_relative
- path::make_preferred
New or improved operations functions include:
- absolute. This replaces the operations function complete(), which is now deprecated. Semantics are now provided for a Windows corner case where the base argument was not an absolute path. Previously this resulted in an exception being thrown.
- create_symlink now supported on both POSIX and Windows.
- read_symlink function added. Supported on both POSIX and Windows. Used to read the contents of a symlink itself.
- resize_file function added. Supported on both POSIX and Windows. Used to shrink or grow a regular file.
- unique_path function added. Supported on both POSIX and Windows. Used to generate a secure temporary pathname.
Support for error reporting via error_code is now uniform throughout the operation functions.
Documentation has been reworked, including re-writes of major portions.
A new Tutorial provides a hopefully much gentler and more complete introduction for new users. Current users might want to review the three sections related to class path.

Deprecated Names and Other Features

See the Deprecated Features page for transition aids that allow much existing code to compile without change using Version 3.

Breaking Changes

Class path
Compiler Support

To ease the transition, Versions 2 and 3 both used to be included in the next several Boost releases. Version 2 was removed in Boost 1.50.0.

Class `path`

Class template basic_path and its specializations are replaced by a single class path. Thus any code, such as overloaded functions, that depend on path and wpath being two distinct types will fail to compile and must be restructured. Restructuring may be as simple as removing one of the overloads, but also might require more complex redesign.
Certain functions now return path objects rather than string or wstring objects:
- root_name()
- root_directory()
- filename()
- stem()
- extension() Not all uses will fail; if the function is being called in a context that accepts a path, all is well. If the result is being used in a context requiring a std::string or std::wstring, then .string() or .wstring() respectively must be appended to the function call.
path::iterator::value_type and path::const_iterator::value_type is path rather than std::basic_string.

Compiler Support

Compilers and standard libraries that do not fully support wide characters and wide character strings (std::wstring) are no longer supported.
Cygwin versions prior to 1.7 are no longer supported because they lack wide string support. Cygwin now compiles only for the POSIX API and path syntax.
MinGW versions not supporting wide strings are no longer supported.
Microsoft VC++ 7.1 and earlier are no longer supported.

Design

Introduction
Problem
Solution
Details
Other Changes
Filesystem Relative Proposal

	Warning
	This section documents thinking early in the V3 development process, and is intended to serve historical purposes. It is not updated to reflect the current state of the library.

	Note
	Some parts of the discussion in this section are spoken from the perspective of the original author of Boost.Filesystem, Beman Dawes. This text is preserved, with minor editorial and formatting changes, for historical reasons.

During the review of Boost.Filesystem.V2 (Internationalization), Peter Dimov suggested that the basic_path class template was unwieldy, and that a single path type that accommodated multiple character types and encodings would be more flexible. Although I wasn't willing to stop development at that time to explore how this idea might be implemented, or to break from the pattern for Internationalization used the C++ standard library, I've often thought about Peter's suggestion. With the advent of C++0x char16_t and char32_t character types, the basic_path class template approach becomes even more unwieldy, so it is time to revisit the problem in light of Peter's suggestion.

Problem

With Filesystem.V2, a path argument to a user defined function that is to accommodate multiple character types and encodings must be written as a template. Do-the-right-thing overloads or template metaprogramming must be employed to allow arguments to be written as string literals. Here's what it looks like:

template<class Path>
void foo(const Path& p);

inline void foo(const path& p)
{
    foo<path>(p);
}

inline void foo(const wpath& p)
{
    foo<wpath>(p);
}

That's really ugly for such a simple need, and there would be a combinatorial explosion if the function took multiple Path arguments and each could be either narrow or wide. It gets even worse if the C++0x char16_t and char32_t types are to be supported.

Solution

Overview:

A single, non-template, class path.
Each member function is a template accommodating the various applicable character types, including user-defined character types.
Hold the path internally in a string of the type used by the operating system API; std::string for POSIX, std::wstring for Windows.

The signatures presented in Problem collapse to simply:

void foo(const path& p);

That's a significant reduction in code complexity. Specification becomes simpler, too. I believe it will be far easier to teach, and result in much more flexible user code.

Other benefits:

All the polymorphism still occurs at compile time.
Efficiency is increased, in that conversions of the encoding, if required, only occur once at the time of creation, not each time the path is used.
The size of the implementation code drops approximately in half and becomes much more readable.

Possible problems:

The combination of member function templates and implicit constructors can result in unclear error messages when the user makes simple commonplace coding errors. This should be much less of a problem with C++ concepts, but in the meantime work continues to restrict over aggressive templates via enable_if/disable_if.

Details

Table 1.2. Encoding Conversions

Host system	char string path arguments	wide string path arguments
Systems with `char` as the native API path character type (i.e. POSIX-like systems)	No conversion.	Conversion occurs, performed by the current path locale's `codecvt` facet.
Systems with `wchar_t` as the native API path character type (i.e. Windows-like systems).	Conversion occurs, performed by the current path locale's `codecvt` facet.	No conversion.

When a class path function argument type matches the operating system's API argument type for paths, no conversion is performed rather than conversion to a specified encoding such as one of the Unicode encodings. This avoids unintended consequences, etc.

Other Changes

Uniform hybrid error handling: The hybrid error handling idiom has been consistently applied to all applicable functions.

Filesystem Relative Proposal

Introduction
Requirements
Issues
Design Decisions
Proposed Wording

Introduction

There have been requests for a Filesystem library relative function for at least ten years. The requested functionality seems simple - given two paths with a common prefix, return the non-common suffix portion of one of the paths such that it is relative to the other path. In terms of the Filesystem library:

path p("/a/b/c");
path base("/a/b");
path rel = relative(p, base);  // the requested function
cout << rel << endl;           // outputs "c"
assert(absolute(rel, base) == p);

If that was all there was to it, the Filesystem library would have had a relative function years ago.

Blocking issues: Clashing requirements, symlinks, directory placeholders (dot, dot-dot), user-expectations, corner cases.

Acknowledgement

A paper by Jamie Allsop, Additions to Filesystem supporting Relative Paths, is what broke my mental logjam. Much of what follows is based directly on Jamie's analysis and proposal. The weakly_canonical function and aspects of the semantic specifications are my contributions. Mistakes, of course, are mine.

Preliminary Implementation

A preliminary implementation is available in the feature/relative2 branch of the Boost Filesystem Git repository.

Requirements

Requirement 1: Some uses require symlinks be followed; i.e. the path must be resolved in the actual file system.
Requirement 2: Some uses require symlinks not be followed; i.e. the path must not be resolved in the actual file system.
Requirement 3: Some uses require removing redundant current directory (dot) or parent directory (dot-dot) placeholders.
Requirement 4: Some uses do not require removing redundant current directory (dot) or parent directory (dot-dot) placeholders since the path is known to be already in normal form.

Issues

Issue 1: What happens if p and base are themselves relative?
Issue 2: What happens if there is no common prefix? Is this an error, the whole of p is relative to base, or something else?
Issue 3: What happens if p, base, or both are empty?
Issue 4: What happens if p and base are the same?
Issue 5: How is the "common prefix" determined?
Issue 6: What happens if portions of p or base exist but the entire path does not exist and yet symlinks need to be followed?
Issue 7: What happens when a symlink in the existing portion of a path is affected by a directory (dot-dot) placeholder in a later non-existent portion of the path?
Issue 8: Overly complex semantics (and thus specifications) in preliminary designs made reasoning about uses difficult.
Issue 9: Some uses never have redundant current directory (dot) or parent directory (dot-dot) placeholders, so a removal operation would be an unnecessary expense although otherwise harmless.

Design Decisions

Provide Separate Lexical and Operational `relative` Functions

Resolves the conflict between requirement 1 and requirement 2 and ensures both requirements are met.

A purely lexical function is needed by users working with directory hierarchies that do not actually exist.

An operational function that queries the current file system for existence and follows symlinks is needed by users working with actual existing directory hierarchies.

Provide Separate Lexical and Operational `proximate` Functions

Although not the only possibility, a likely fallback when the relative functions cannot find a relative path is to return the path being made relative. As a convenience, the proximate functions do just that.

Add Lexical Functions as `path` Member Functions

The Filesystem library is unusual in that it has several functions with both lexical (i.e. cheap) and operational (i.e. expensive due to file system access) forms with differing semantics. It is important that users choose the form that meets their application's specific needs. The library has always made the distinction via the convention of lexical functions being members of class path, while operational functions are non-member functions. The lexical functions proposed here also use the name prefix lexically_ to drive home the distinction.

For the contrary argument, see Sutter and Alexandrescu, C++ Coding Standards, 44: "Prefer writing nonmember nonfriend functions", and Meyers, Effective C++ Third Edition, 23: "Prefer non-member non-friend functions to member functions."

Provide a Non-member Function `lexically_normal` Returning a Normal Form Path

Enables resolution of requirement 3 and requirement 4 in a way consistent with issue 9. Is a contributor to the resolution of issue 8.

"Normalization" is the process of removing redundant current directory (dot), parent directory (dot-dot), and directory separator elements.

Normalization is a byproduct the current canonical function. But for the path returned by the proposed weakly_canonical function, only any leading canonic portion is in canonical form. So any trailing portion of the returned path has not been normalized.

Jamie Allsop has proposed adding a separate normalization function returning a path, and I agree with him.

Boost.Filesystem has a deprecated non-const normalization function that modifies the path, but I agree with Jamie that a function returning a path is a better solution.

Provide a `weakly_canonical` Operational Function

Resolves issue 6, issue 7, issue 9, and is a contributor to the resolution of issue 8.

The operational function weakly_canonical(p) returns a path composed of canonical(x)/y, where x is a path composed of the longest leading sequence of elements in p that exist, and y is a path composed of the remaining trailing non-existent elements of p if any. "weakly" refers to weakened existence requirements compared to the existing canonical function.

Having weakly_canonical as a separate function, and then specifying the processing of operational relative arguments in terms of calls to weakly_canonical makes it much easier to specify the operational relative function and reason about it. The difficulty of reasoning about operational relative semantics before the invention of weakly_canonical was what led to its initial development.
Having weakly_canonical as a separate function also allows use in other contexts.
Specifying the return be in normal form is an engineering trade-off to resolve issue 7 in a way that just works for most use cases.
Specifying normative encouragement to not perform unneeded normalization is a reasonable resolution for issue 9.

Resolve Issues in Ways that "Just Work" for Users

Resolves issues 1, 2, 3, 4, 6, and 7. Is a contributor to the resolution of issue 8.

The "just works" approach was suggested by Jamie Allsop. It is implemented by specifying a reasonable return value for all of the "What happens if..." corner case issues, rather that treating them as hard errors requiring an exception or error code.

Specify `lexically_relative` in Terms of `std::mismatch`

Resolves issue 5. Is a contributor to the resolution of issue 8.

Specify `relative` in Terms of `weakly_canonical`

Is a contributor to the resolution of [filesystem.v3.design.relative-proposal.issues.issue-8 issue 8].

Covers a wide range of uses cases since a single function works for existing, non-existing, and partially existing paths.
Works correctly for partially existing paths that contain symlinks.

Specify `relative` in Terms of `lexically_relative`

Is a contributor to the resolution of issue 5 and issue 8.

If would be confusing to users and difficult to specify correctly if the two functions had differing semantics:

When either or both paths are empty.
When all elements of the two paths match exactly.
Because different matching algorithms were used.
Because although the same matching algorithm was used, it was applied in different ways.

These problems are avoided by specifying operational relative in terms of lexical relative after preparatory calls to operational functions.

Proposed Wording

	Note
	"Overview:" sections below are non-normative experiments attempting to make the normative reference specifications easier to grasp.

Define Normal Form

A path is in normal form if it has no redundant current directory (dot) or parent directory (dot-dot) elements. The normal form for an empty path is an empty path. The normal form for a path ending in a directory-separator that is not the root directory is the same path with a current directory (dot) element appended.

	Note
	The last sentence above is not necessary for POSIX-like or Windows-like operating systems, but supports systems like OpenVMS that use different syntax for directory and regular-file names.

New Class `path` Member Functions

Synopsis

path lexically_normal() const;
path lexically_relative(const path& base) const;
path lexically_proximate(const path& base) const;

Specification

path lexically_normal() const;

Overview: Returns *this with redundant current directory (dot), parent directory (dot-dot), and directory-separator elements removed.

Returns: *this in normal form.

Remarks: Uses operator/= to compose the returned path.

[Example:
assert(path("foo/./bar/..").lexically_normal() == "foo");
assert(path("foo/.///bar/../").lexically_normal() == "foo/.");
The above assertions will succeed. On Windows, the returned path's directory-separator characters will be backslashes rather than slashes, but that does not affect path equality. —end example]

path lexically_relative(const path& base) const;

Overview: Returns *this made relative to base. Treats empty or identical paths as corner cases, not errors. Does not resolve symlinks. Does not first normalize *this or base.

Remarks: Uses std::mismatch(begin(), end(), base.begin(), base.end()), to determine the first mismatched element of *this and base. Uses operator== to determine if elements match.

Returns:

path() if the first mismatched element of *this is equal to begin() or the first mismatched element of base is equal to base.begin(), or

path(".") if the first mismatched element of *this is equal to end() and the first mismatched element of base is equal to base.end(), or

An object of class path composed via application of operator/=("..") for each element in the half-open range [first mismatched element of base, base.end()), and then application of operator/= for each element in the half-open range [first mismatched element of *this, end()).

[Example:
assert(path("/a/d").lexically_relative("/a/b/c") ==  "../../d");
assert(path("/a/b/c").lexically_relative("/a/d") ==  "../b/c");
assert(path("a/b/c").lexically_relative("a") == "b/c");
assert(path("a/b/c").lexically_relative("a/b/c/x/y") ==  "../..");
assert(path("a/b/c").lexically_relative("a/b/c") ==  ".");
assert(path("a/b").lexically_relative("c/d") ==  "");
The above assertions will succeed. On Windows, the returned path's directory-separators will be backslashes rather than forward slashes, but that does not affect path equality. —end example]

[Note: If symlink following semantics are desired, use the operational function relative —end note]

[Note: If normalization is needed to ensure consistent matching of elements, apply lexically_normal() to *this, base, or both. —end note]

path lexically_proximate(const path& base) const;

Returns: If the value of lexically_relative(base) is not an empty path, return it. Otherwise return *this.

[Note: If symlink following semantics are desired, use the operational function proximate —end note]

[Note: If normalization is needed to ensure consistent matching of elements, apply lexically_normal() to *this, base, or both. —end note]

New operational functions

Synopsis

path weakly_canonical(const path& p);
path weakly_canonical(const path& p, system::error_code& ec);
path relative(const path& p, system::error_code& ec);
path relative(const path& p, const path& base=current_path());
path relative(const path& p, const path& base, system::error_code& ec);
path proximate(const path& p, system::error_code& ec);
path proximate(const path& p, const path& base=current_path());
path proximate(const path& p, const path& base, system::error_code& ec);

Specification

path weakly_canonical(const path& p);
path weakly_canonical(const path& p, system::error_code& ec);

Overview: Returns p with symlinks resolved and the result normalized.

Returns: A path composed of the result of calling the canonical function on a path composed of the leading elements of p that exist, if any, followed by the elements of p that do not exist, if any.

Postcondition: The returned path is in normal form.

Remarks: Uses operator/= to compose the returned path. Uses the status function to determine existence.

Remarks: Implementations are encouraged to avoid unnecessary normalization such as when canonical has already been called on the entirety of p.

Throws: As specified in Error reporting.

path relative(const path& p, system::error_code& ec);

Returns: relative(p, current_path(), ec).

Throws: As specified in Error reporting.

path relative(const path& p, const path& base=current_path());
path relative(const path& p, const path& base, system::error_code& ec);

Overview: Returns p made relative to base. Treats empty or identical paths as corner cases, not errors. Resolves symlinks and normalizes both p and base before other processing.

Returns: weakly_canonical(p).lexically_relative(weakly_canonical(base)). The second form returns path() if an error occurs.

Throws: As specified in Error reporting.

path proximate(const path& p, system::error_code& ec);

Returns: proximate(p, current_path(), ec).

Throws: As specified in Error reporting.

path proximate(const path& p, const path& base=current_path());
path proximate(const path& p, const path& base, system::error_code& ec);

Returns: weakly_canonical(p).lexically_proximate(weakly_canonical(base)). The second form returns path() if an error occurs.

Throws: As specified in Error reporting.