Aquileo | Recent changes to wiki

Aquileo | AdminChanges modified by Joseph Koshy

Tue, 09 May 2023 07:04:31 -0000

--- v16
+++ v17
@@ -2,6 +2,7 @@

 See also: the project-level [audit page](https://sourceforge.net/p/elftoolchain/admin/audit/) provided by  SourceForge.

+  * 2020-03-31 jkoshy: Set SF.Net project feature list to: 'ELF', 'DWARF'.
   * 2019-12-15 jkoshy: Enable [HTTPS hosting](https://sourceforge.net/p/forge/documentation/Convert%20your%20website%20to%20HTTPS/).  The project website may now be accessed at https://elftoolchain.sourceforge.io/.
   * 2016-02-18 jkoshy: Version [v0.7.1](Releases-Release0.7.1) released.
   * 2016-01-19 jkoshy: Change the default milestone for tickets to 'RELEASE_1_0'.

Aquileo | libtask modified by Joseph Koshy

Sun, 18 Dec 2022 21:23:24 -0000

libtask

A simple task management API, intended for tools that need to sequence (potentially parallel) computations, like test runners.

Aquileo | Home modified by Joseph Koshy

Sun, 18 Dec 2022 21:20:31 -0000

--- v33
+++ v34
@@ -282,6 +282,14 @@
 kaiwang27
 libpe/
 
+
+
+libtask
+A  simple task management facility
+planning
+jkoshy
+
+For use in tools that need to sequence potentially parallel computations.
 
 
 Test Suites

Aquileo | libpe modified by Joseph Koshy

Sun, 18 Dec 2022 21:12:58 -0000

libpe

A library for handling the Microsoft® 'Portable Executable' file format.

Aquileo | Home modified by Joseph Koshy

Sun, 18 Dec 2022 21:11:45 -0000

--- v32
+++ v33
@@ -276,7 +276,7 @@
 For use in an optimizing linker, debugger and assember, program instrumentation tools, and the like. 
 
 
-libpe
+libpe
 A library for reading and writing Portable Executable™objects
 alpha
 kaiwang27

Aquileo | Home modified by Joseph Koshy

Tue, 26 Apr 2022 06:28:54 -0000

--- v31
+++ v32
@@ -1,4 +1,8 @@
 # The ELF Tool Chain Project
+
+## About This Project
+
+We develop [BSD licensed](http://en.wikipedia.org/wiki/BSD_licenses) implementations of essential development tools and libraries for building and analyzing [ELF](http://en.wikipedia.org/wiki/Executable_and_Linkable_Format) based program images.

 ## Quick Links

@@ -15,9 +19,7 @@
   * **2012/09/24**: [Release 0.6.1](Releases-Release0.6.1) is now available for [download](https://sourceforge.net/projects/elftoolchain/files/Sources/elftoolchain-0.6.1/). 
   * **2011/11/16**: [Release 0.5.1](Releases-Release0.5.1) is now available for [download](https://sourceforge.net/projects/elftoolchain/files/Sources/elftoolchain-0.5.1/). 

-## About This Project
-
-We maintain [BSD licensed](http://en.wikipedia.org/wiki/BSD_licenses) implementations of essential development tools, and libraries for building and analyzing [ELF](http://en.wikipedia.org/wiki/Executable_and_Linkable_Format) based program images. 
+

 ### Supported Operating Systems

Aquileo | Home modified by Joseph Koshy

Mon, 08 Mar 2021 22:24:07 -0000

--- v30
+++ v31
@@ -41,13 +41,22 @@
   * Our code is licensed under a [liberal license](http://en.wikipedia.org/wiki/BSD_licenses). 
   * As a project, we strive to make our decisions in a transparent and inclusive manner.

-### What are the shortcomings of this project?
+### Users
+
+Elftoolchain code is usually imported by operating system projects into their respective source trees.  Operating system projects known to be using Elftoolchain code include:
+
+* [FreeBSD](https://svnweb.freebsd.org/base/head/contrib/elftoolchain/)
+* [NetBSD](http://cvsweb.netbsd.org/bsdweb.cgi/src/external/bsd/elftoolchain/)
+* [OpenBSD](https://github.com/openbsd/src/tree/master/lib/libelf)
+* [RTEMS](https://devel.rtems.org/browser/rtems-tools/rtemstoolkit/elftoolchain/libelf)
+
+### Caveats

   * Our code is relatively new, it has not seen the level of adoption that the GNU binutils code base has. 
   * Our code base only deals with ELF objects; other object file formats are not well supported. 
   * Our development team is small, comprising of volunteers. 

-### How do I help?
+### How to help

   * Please join the [mailing list](http://sourceforge.net/mailarchive/forum.php?forum_name=elftoolchain-developers); let us know how you are using our code and how your experience has been. 
   * Please review our code and documentation; [let us know](http://sourceforge.net/apps/trac/elftoolchain/newticket) if you find a bug.

Aquileo | BuildAutomation modified by Joseph Koshy

Fri, 05 Feb 2021 18:44:36 -0000

--- v20
+++ v21
@@ -2,16 +2,14 @@

 This wiki page describes a proposed build system for the [Elftoolchain project](Home).

-This build system is intended to be "*P2P Transparent*" (the precise definition of this term is given further down this article).
+This build system is intended to be *"P2P Transparent"* (the precise definition of this term is given further down).

 ## Goals

-This build system allows developers to **share their build cycles**, while also being assured of the **integrity of their builds**.
-
 The major goals for the system are:

--   To implement repeatable & hermetic (i.e., fully deterministic) builds from source code.
--   To support "build transparency": to offer the ability to verify that an output artifact (say the ISO image for an installer) had been verifiably built from a specific set of human-reviewed source files.
+-   To support repeatable & hermetic (i.e., fully deterministic) builds from source code.
+-   To support "build transparency": the ability to verify that an output artifact (say the ISO image for an installer) had been verifiably built from a specific set of human-reviewed source files.
 -   To support cross-building of source code.
 -   To allow developers to securely share build cycles with each other (peer-to-peer sharing of their build effort).
 -   To be frugal in its machine resource usage.
@@ -19,11 +17,21 @@

 ## Terminology
 ### Repeatable Builds
-#### Local Repeatability
-
-For the purposes of this document, a build is said to *locally repeatable* when two attempts at building the same source tree on a given machine would result in the 'same' bits being generated.  (I'm using 'same' here as shorthand for "logically indistinguishable by a subsequent build step").
-
-Many compilation processes do not guarantee local repeatability out of the box.  Local non-deterrminism may be caused by the presence of timestamps in the output of a build step, embedded temporary filenames in generated output, output differences caused by unstable orderings of inputs, the use of special macros like `__DATE__` and `__TIME__`, and the like.
+
+**Local Repeatability**
+
+We consider a build to  be locally repeatable when two attempts at building the same source tree on a given machine result in the 'same' effective bits being generated.
+
+The word 'same' is used here to mean "logically indistinguishable when used by a subsequent build step". 
+
+Many compilation processes do not guarantee local repeatability out of the box.   Some causes of local non-repeatibility include:
+
+*   The presence of compilation timestamps in the output of a build step,
+*   Embedded names of temporary files in generated output,
+*   Differences caused by unstable orderings of inputs,
+*   The use of special macros like `__DATE__` and `__TIME__`,
+
+and the like.

 Prior work in making builds locally repeatable includes:

@@ -32,34 +40,38 @@
 -  [Reproducible Builds in NetBSD](https://blog.netbsd.org/tnf/entry/netbsd_fully_reproducible_builds), from the [NetBSD](https://www.netbsd.org/) project.
 -  [Deterministic Builds](https://blog.torproject.org/deterministic-builds-part-one-cyberwar-and-global-compromise) at the [Tor Project](https://www.torproject.org/).

-Some Elftoolchain tools support locally repeatable output (usually specified using the `-D` option).
-
-A locally repeatable build allows the build process to be treated as a *pure function* mapping a set of source artifacts to a set of built artifacts.  Local repeatability would be the first step towards sharing build cycles across multiple developers.
-
-#### Cross-Machine Repeatability a.k.a. Global Reproducibility
-
-In a globally reproducible build, builds on different machines generate the 'same' (i.e., logically indistinguishable) output.  Two different machines are permitted to have different build execution graphs, as long as the final outputs are the 'same'.  For example, the build on one machine could have additional instrumentation interleaved between its build steps, or could use tools that differ from the other.
-
-Some sources of "non-local" non-determinism include: the use of embedded (absolute) pathnames in generated output, build time dependencies on the time zone of the build machine, and so on.
-
-Global reproducibility is needed before we can meaningfully share build cycles across machines.
+Some Elftoolchain tools produce locally repeatable output (this mode is usually specified using the `-D` option).
+
+A locally repeatable build permits the build process to be treated as a *pure function* mapping a set of sources to a set of built artifacts.  
+
+Local repeatability is the first step towards sharing build cycles across multiple developers.
+
+**Cross-Machine Repeatability** a.k.a. **Global Reproducibility**
+
+In a globally reproducible build, builds on different machines generate the 'same' (i.e., logically indistinguishable) output.  Two different machines are permitted to have different build execution graphs, as long as the final outputs are the 'same'.  For example, the build on one machine could have additional instrumentation interleaved between its build steps, or could use tools that differ from the other, provided the end result is the 'same'.
+
+Some sources of "non-local" non-determinism include:
+
+*   The use of embedded (absolute) pathnames in generated output,
+*   Build time dependencies on the time zone of the build machine,
+
+and so on.
+
+Global reproducibility is needed before we can share build cycles across machines.

 ### Hermetic Builds

-In a *hermetic build* every input to the build process is fully specified.  These inputs include:
+In a hermetic build all inputs to the build process is fully specified.  These include:

 - The version of the compilation tools in use.
 - The configuration used when invoking compilation tools: invocation flags, linker scripts, and so on.
-- Each and every module that a given step in the build depends on.  This includes the headers, libraries and configuration traditionally considered to be part of the "base system".
-- And of course the notional "source code" being built.
+- Each and every input that a given step in the build depends on.  This includes the headers, libraries and configuration traditionally considered to be part of the base system, apart from the source code being used as input.

 A hermetic build process guarantees that the build does not depend upon inputs that are "external" in some fashion (and are thus uncontrolled).

-One way to achieve a hermetic build would be have the build framework invoke the individual steps in a build in an controlled environment that contains precisely those files that were explicitly determined to be inputs for the build step (e.g. the build step would be run in a [chroot](https://en.wikipedia.org/wiki/Chroot), a [jail](https://www.freebsd.org/doc/handbook/jails.html) or a symlink tree to a set of read-only inputs).
-
-If build configuration is tracked in version control alongside the rest of the source tree, and if the source code to the compilation tools is also part of the source tree (e.g. as in [NetBSD](https://www.netbsd.org/docs/guide/en/part-compile.html)) then the build can be made *hermetic through time*.  With a hermetic-through-time build we would be able to roll back a source tree to a prior revision and be able to build exactly the same output using the exact same tooling as would have been originally built at that revision.
-
-###  Peer To Peer Builds
+If the source code to the compilation tools is part of the source tree being built (e.g. as is true for [NetBSD](https://www.netbsd.org/docs/guide/en/part-compile.html)) then the build can be made *hermetic through time*.  With a hermetic-through-time build we would be able to roll back a source tree to a prior revision and be able to reconstruct binaries using the exact toolchain that had been originally used at that revision.
+
+**Peer To Peer Builds**

 Once builds are *hermetic* and *globally reproducible*, we would be able to share builds across build peers.

@@ -71,39 +83,39 @@

 Related work: [Bazel][bazel] supports caching of build results using a (centralized) [remote cache](https://docs.bazel.build/versions/master/remote-caching.html).

-#### Herd Immunity and Decentralized Verification
-
-Peer-to-peer builds could also be used to implement a form of "herd immunity".  Once we have multiple builders building the same source tree and sharing metadata about their builds, it would become possible to detect output artifacts that had been tampered with.
+**Herd Immunity** and **Decentralized Verification**
+
+Peer-to-peer builds could also be used to implement a form of herd immunity.  Once we have multiple builders building the same source tree and sharing metadata about their builds, it would become possible to detect output artifacts that had been tampered with.

 This tamper detection protocol could be used to verify, in a decentralized fashion, that the final artifact generated by a build (say the ISO image containing an OS installer) had not been tampered with locally.

-### Auditable Builds
-
-This kind of build process generates a (cryptographically signed) trail of the inputs, outputs and tools used for each build step.
+**Auditable Builds**
+
+This kind of build generates a (cryptographically signed) trail of the inputs, outputs and tools used for each build step.

 This trail could be used to verify the build 'end to end'.

-### P2P Auditable Builds
+**P2P Auditable Builds**

 In this document this term means an extension of auditable builds that share outputs and build trails in a P2P fashion.

-### Vetted Source
+**Vetted Source**

 Source code that is linked to a 'is-reviewed' bit issued by a human.

-### Vetted Builds
+**Vetted Builds**

 These are builds whose source code inputs can be proven to have been vetted by one or more humans.

-### Transparent Builds
-
-I am using this term to mean an *auditable* and *vetted* build.
-
-### P2P Transparent Build
+**Transparent Builds**
+
+This is a build that is both *auditable* and *vetted*.
+
+**P2P Transparent Build**

 This is a *transparent* build which uses a P2P build-sharing backend.

-### Cross-Organisational Verifiability
+**Cross-Organisational Verifiability**

 Sometimes a software build would take for its input one or more binary blobs built by external organisations, for which source code is not available.

@@ -113,7 +125,7 @@

 TODO(jkoshy): Look for existing standards and protocols implementing cross-organisational verifiability, if these exist.

-### The Build Configuration Language
+## The Build Configuration Language

 BSD operating systems are built from source using [BSD make][make] (e.g., the [FreeBSD makeworld](https://www.freebsd.org/doc/handbook/makeworld.html) build process, or by using an additional tool that drives [make][], such as in [NetBSD's build.sh](https://www.netbsd.org/docs/guide/en/part-compile.html)).

@@ -123,7 +135,7 @@

 At this point of time I am inclined to use a notation that is close to S-expressions, for the reasons outlined in some of the sections below.

-#### Tooling friendliness of the notation
+### Tooling friendliness of the notation

 The build configuration language should be tooling-friendly. Large projects almost always need such tooling support. This means that the build configuration language should be easy to parse and modify programmatically.

@@ -156,7 +168,7 @@
   deps: (...))
 ```

-#### Support for modularity
+### Support for modularity

 The build system should support the structuring of project code into logical modules.

@@ -164,19 +176,19 @@

 [Make][], in contrast uses a single global namespace, with its attendant lack of modularity.  Large projects then end up using recursive invocations of `make`, which all the drawbacks documented in  [Recursive make Considered Harmful](http://aegis.sourceforge.net/auug97.pdf) (Peter Miller, 2008).

-#### Separation of concerns
+### Separation of concerns

 We would like the configuration used to specify build & test scenarios (i.e., different compilation modes: `-c optimize` vs `-c debug` or running tests under `valgrind` vs running them normally) to be separate from the configuration that specifies the dependencies between input and output artifacts.  These two aspects of build configuration are intertwined in [make][].

 We want users to be able to easily define their own scenarios for building and testing code.

-#### Multiple logically connected build outputs per directory
+### Multiple logically connected build outputs per directory

 The BSD `/usr/share/mk/*.mk` framework currently works best when a single sub-directory in a source tree builds one program or one library. This restriction can be fairly limiting.

 We would like our build system to allow test suites to be co-located with code that they test.   The build system should allow multiple logically related libraries and binaries to be built in a single source directory.

-#### Expressivity
+### Expressivity

 Builds for large projects would need a language for writing "build macros" and the like.

@@ -186,35 +198,35 @@

 If we go the S-expression route, then a declarative subset of R*n*RS Scheme may be a natural choice for the extension language.  

-### Other Features Desired For The Build System
-
-#### Offline Operation
+## Other Features Desired For The Build System
+
+### Offline Operation

 The build system should be able to work in 'offline' mode, without needing external network access to function (although builds would be entirely local in this case).

 This feature would be useful for developers in areas of poor network connectivity.

-#### Frugality
+### Frugality

 The build system should be able to work effectively in a resource constrained host environment (such as a laptop computer).

 [Bazel][] in particular seems to need a lot of memory when building projects.

-#### Cross-platform Builds
+### Cross-platform Builds

 The build system should support building code on a variety of (non-native) machine architectures and operating systems.

 Cross building an operating system often involves compiling the cross compiler itself.  The build tool should be able to handle these kinds of scenarios efficiently.

-#### Building With Local Changes
+### Building With Local Changes

 The build system should be able to work efficiently on a source tree with a small number of local changes, sharing the build of the unmodified parts of the source tree with its peers.

-#### Easing (Build) Portability
+### Easing (Build) Portability

 New target architectures should be easy to support, either through cross compilation, or by invoking the necessary build steps "remotely" using a small, portable helper that runs on the target hardware.

-#### Support for Bootstrapping
+### Support for Bootstrapping

 The build tool should be usable when bootstrapping an operating system from source.

Aquileo | Home modified by Joseph Koshy

Sun, 31 May 2020 11:12:23 -0000

--- v29
+++ v30
@@ -17,7 +17,7 @@

 ## About This Project

-We maintain [BSD licensed](http://en.wikipedia.org/wiki/BSD_licenses) implementations of essential compilation tools and libraries for building and analyzing [ELF](http://en.wikipedia.org/wiki/Executable_and_Linkable_Format) based program images. 
+We maintain [BSD licensed](http://en.wikipedia.org/wiki/BSD_licenses) implementations of essential development tools, and libraries for building and analyzing [ELF](http://en.wikipedia.org/wiki/Executable_and_Linkable_Format) based program images. 

 ### Supported Operating Systems

@@ -35,7 +35,8 @@

 This project provides tools for program development. Notably: 

-  * Our code is structured as a set of tools written using a set of well-documented libraries. This approach comes in useful when custom tools for manipulating ELF programs and shared libraries are desired. 
+  * Our code is structured as a set of tools written using a set of well-documented libraries. These libraries are useful for creating custom tools for manipulating ELF programs and shared libraries.
+  * We also provide out-of-the-box implementations of common program development tools built over these libraries.
   * We strive to keep to the traditional [BSD standard](ElfToolChainAdvocacy) for quality in code and documentation. We strive to 'Do The Right Thing'. 
   * Our code is licensed under a [liberal license](http://en.wikipedia.org/wiki/BSD_licenses). 
   * As a project, we strive to make our decisions in a transparent and inclusive manner.

Aquileo | Home modified by Joseph Koshy

Sun, 31 May 2020 11:04:20 -0000

--- v28
+++ v29
@@ -43,7 +43,7 @@
 ### What are the shortcomings of this project?

   * Our code is relatively new, it has not seen the level of adoption that the GNU binutils code base has. 
-  * Our code base only deals with ELF objects; other object file formats are not supported. 
+  * Our code base only deals with ELF objects; other object file formats are not well supported. 
   * Our development team is small, comprising of volunteers. 

 ### How do I help?