1. 03 May, 2019 27 commits
    • Wez Furlong's avatar
      fbcode_builder: getdeps: add NopBuilder · b4135123
      Wez Furlong authored
      Summary:
      this could do with a better name; the NopBuilder doesn't actually
      build anything, but instead copies some files to the destination location.
      
      This is used together with eg: cmake to install pre-built binaries downloaded
      from a tarball.
      
      Reviewed By: simpkins
      
      Differential Revision: D14691015
      
      fbshipit-source-id: a938e977aa4ec5a664bdb8085ff708319a204594
      b4135123
    • Wez Furlong's avatar
      fbcode_builder: getdeps: add Boost builder · b09be4e0
      Wez Furlong authored
      Summary:
      the boost builder knows how to perform the non-standard
      configure and build for boost.
      
      Ideally we'd just build this statically and be happy but there are
      some nuances I've observed while building on different platforms:
      
      * One of our projects (thrift or wangle) explicitly requests static
        boost linkage for reasons unspecified
      * on darwin the install_name is broken when building dynamic libs
      
      For the sake of expediency in getting getdeps up and running, the
      solution for the moment is to build static on posix systems and
      build both static and shared on windows systems.
      
      Reviewed By: simpkins
      
      Differential Revision: D14691009
      
      fbshipit-source-id: 634770a6f53c3ada42d1877cc6c3dacc6eed7d18
      b09be4e0
    • Wez Furlong's avatar
      fbcode_builder: getdeps: add OpenSSL builder · 73da623a
      Wez Furlong authored
      Summary:
      the openssl builder knows how to perform the non-standard
      configuration and build steps to build openssl.
      
      On Linux systems the manifests for our projects don't mention
      openssl, causing them to pick up the system openssl.
      
      On Mac, apple don't ship openssl headers so we need to build our own.
      On Windows there is no standard openssl installation so we also need
      to build our own.
      
      As a result, this builder only works on windows and mac.
      
      Reviewed By: simpkins
      
      Differential Revision: D14691010
      
      fbshipit-source-id: 9f8979f9eaeb5209c290cf4f43c97c0cb43d13a2
      73da623a
    • Wez Furlong's avatar
      fbcode_builder: getdeps: add NinjaBootstrap builder · 5a16dcf3
      Wez Furlong authored
      Summary:
      this builder is used to bootstrap the ninja build tool.
      
      On Windows and mac the manifest for ninja is set to download a pre-built executable.
      While pre-built executables are available for linux they aren't portable enough
      for our purposes so we need to be able to build it for ourselves.
      
      Reviewed By: simpkins
      
      Differential Revision: D14690992
      
      fbshipit-source-id: b60fd02ad04f58dc7c2931280341791270609737
      5a16dcf3
    • Wez Furlong's avatar
      fbcode_builder: getdeps: add CMakeBuilder · 1f158eff
      Wez Furlong authored
      Summary:
      the cmake builder knows how to use cmake to configure a build
      for (preferably) and out-of-src build.  The `cmake.defines` section of
      the manifest is used to pass `-Dkey=value` options to the cmake configure
      command line.
      
      We prefer to use `ninja` to execute the build so that we can use more
      cores than 1 on Windows and just for consistency across platforms
      with mac and linux.
      
      Reviewed By: simpkins
      
      Differential Revision: D14690998
      
      fbshipit-source-id: 8102e8b4a47da515ca001772788ed0e5f2645ad7
      1f158eff
    • Wez Furlong's avatar
      fbcode_builder: getdeps: add AutoconfBuilder · 840ee5bb
      Wez Furlong authored
      Summary:
      the autoconf builder performs an out-of-source build using
      the autoconf suite to configure the build rules.
      
      The `autoconf.args` section from the manifest is passed to the `./configure`
      command line.
      
      If an `autogen.sh` script is present then it will be used to regenerate
      a missing `configure` script, otherwise we'll try `autoreconf`.
      
      Reviewed By: simpkins
      
      Differential Revision: D14691002
      
      fbshipit-source-id: ab8cceafb833dab513d5a50c65f4c895a4f40047
      840ee5bb
    • Wez Furlong's avatar
      fbcode_builder: getdeps: add MakeBuilder · 06b37c9d
      Wez Furlong authored
      Summary:
      the make builder runs `make` in the source directory.
      The `make.args` section from the manifest is passed as arguments
      to the `make` invocation.
      
      Reviewed By: simpkins
      
      Differential Revision: D14690996
      
      fbshipit-source-id: 180d657ad05f0c0266a8c1d30979d8d1473958c9
      06b37c9d
    • Wez Furlong's avatar
      fbcode_builder: getdeps: add BuilderBase class · 1274933e
      Wez Furlong authored
      Summary:
      a builder knows how to build and install a project.
      
      Later diffs add concrete implementations of the BuilderBase
      
      Reviewed By: simpkins
      
      Differential Revision: D14691018
      
      fbshipit-source-id: 89b14614b5160353cd7e59f27037afcdf6229eb7
      1274933e
    • Wez Furlong's avatar
      fbcode_builder: getdeps: add list-deps subcommand · 1034b669
      Wez Furlong authored
      Summary:
      While the command isn't necessarily super useful
      on its own, it does show that the plumbing for walking the
      deps is functioning, and that is important when it comes
      to building.
      
      The output lists the projects in the order that they
      would be built.
      
      The `fetch` command has been augmented to add a `--recursive`
      flag that uses the same mechanism to recursively fetch
      the dependencies.
      
      Reviewed By: simpkins
      
      Differential Revision: D14691004
      
      fbshipit-source-id: b00bf6ad4742f8bb0a70698f71a5fe03d6a1f453
      1034b669
    • Wez Furlong's avatar
      fbcode_builder: getdeps: add fetch subcommand · 655e8e63
      Wez Furlong authored
      Summary:
      Adds a command that can be used to trigger a fetch for a project
      
      ```
      $  ./opensource/fbcode_builder/getdeps.py fetch zstd
      Cloning https://github.com/facebook/zstd.git...
       ---
      + git clone --depth=100 https://github.com/facebook/zstd.git /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/repos/github.com-facebook-zstd.git
      Cloning into '/data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/repos/github.com-facebook-zstd.git'...
      remote: Enumerating objects: 3816, done.
      remote: Counting objects: 100% (3816/3816), done.
      remote: Compressing objects: 100% (1415/1415), done.
      remote: Total 3816 (delta 2556), reused 3312 (delta 2288), pack-reused 0
      Receiving objects: 100% (3816/3816), 2.93 MiB | 9.59 MiB/s, done.
      Resolving deltas: 100% (2556/2556), done.
      Updating /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/repos/github.com-facebook-zstd.git -> v1.3.8
       ---
      + git -C /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/repos/github.com-facebook-zstd.git fetch origin v1.3.8
      From https://github.com/facebook/zstd
       * tag               v1.3.8     -> FETCH_HEAD
       ---
      + git -C /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/repos/github.com-facebook-zstd.git checkout FETCH_HEAD
      Note: checking out 'FETCH_HEAD'.
      
      You are in 'detached HEAD' state. You can look around, make experimental
      changes and commit them, and you can discard any commits you make in this
      state without impacting any branches by performing another checkout.
      
      If you want to create a new branch to retain commits you create, you may
      do so (now or later) by using -b with the checkout command again. Example:
      
        git checkout -b <new-branch-name>
      
      HEAD is now at 470344d Merge pull request #1479 from facebook/visualTest
       ---
      + git -C /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/repos/github.com-facebook-zstd.git submodule update --init
      ```
      
      Reviewed By: simpkins
      
      Differential Revision: D14691008
      
      fbshipit-source-id: 3afa391360518a08ebd6ff97f5b8b4993f10c4e8
      655e8e63
    • Wez Furlong's avatar
      fbcode_builder: getdeps: add SimpleShipitTransformer fetcher · 1c6eb0c9
      Wez Furlong authored
      Summary:
      This fetcher knows how to transform a 1st party project
      from fbsource into approximately the same shape as ShipIt produces
      for the github repo mirrors.  It does this by reading shipit
      mapping information from the manifest file for the project.
      
      Since this fetcher uses data in the manifest and is implemented
      directly in the getdeps codebase, it is suitable for iterating
      on the opensource builds directly out of fbsource on both devservers
      and laptops inside FB.
      
      Reviewed By: simpkins
      
      Differential Revision: D14691012
      
      fbshipit-source-id: 05f68a7be64a2e465937b24b8825d25d3348ed13
      1c6eb0c9
    • Wez Furlong's avatar
      fbcode_builder: getdeps: add ShipitTransformer fetcher · a64ea0a2
      Wez Furlong authored
      Summary:
      This fetcher knows how to take a 1st party FB project
      from fbsource and transform it to the same shape as our github
      repos using ShipIt.  The transformation creates a transformed
      mirror of the code in scratch space (rather than mutating fbsource
      directly).
      
      This can only be used in environments where shipit is available.
      A later diff implements an alternative that works in more environments.
      
      Reviewed By: simpkins
      
      Differential Revision: D14691013
      
      fbshipit-source-id: 539e307755c9fc0a098a235868ab622652061494
      a64ea0a2
    • Wez Furlong's avatar
      fbcode_builder: getdeps: add ArchiveFetcher · d84ca6e6
      Wez Furlong authored
      Summary:
      this fetcher knows how to download a URL that references
      an archive and extract the sources from it.  Compressed tarballs
      and zipfiles are supported.
      
      Reviewed By: simpkins
      
      Differential Revision: D14690994
      
      fbshipit-source-id: ffbbd1d9479b0d5aa3e5cd13fe11a560e9d01c2a
      d84ca6e6
    • Wez Furlong's avatar
      fbcode_builder: getdeps: add GitFetcher · 755d61bb
      Wez Furlong authored
      Summary: this fetcher knows how to check out sources via git.
      
      Reviewed By: simpkins
      
      Differential Revision: D14691000
      
      fbshipit-source-id: 60f1ffbfed7b32a019aef6aa70ae0903f2782451
      755d61bb
    • Wez Furlong's avatar
      fbcode_builder: getdeps: define the Fetcher interface · 0263a36f
      Wez Furlong authored
      Summary:
      Fetchers are used to fetch a source directory from a
      source defined in a manifest file.
      
      More details can be found in comments on the various methods.
      
      The Manifest class offers a create_fetcher method for constructing
      an appropriate fetcher.  This is just a stub in this commit, but
      will expand over the course of the next few diffs as concrete
      fetcher instances are added.
      
      Reviewed By: simpkins
      
      Differential Revision: D14691001
      
      fbshipit-source-id: 8c9038eacae3345e9403d5d1304bf979a9ee1555
      0263a36f
    • Wez Furlong's avatar
      fbcode_builder: getdeps: add build options · e2b81155
      Wez Furlong authored
      Summary:
      The build options class contains some environmental and
      build related information that will be passed down to fetcher
      and builder objects that will be introduced in later diffs.
      
      Reviewed By: simpkins
      
      Differential Revision: D14691007
      
      fbshipit-source-id: e8fe7322f590667ac28a5a3925a072056df0b3e3
      e2b81155
    • Wez Furlong's avatar
      fbcode_builder: getdeps: add platform type tuple helper · b499111d
      Wez Furlong authored
      Summary:
      This will feed into the manifest context for system
      dependent manifest sections.  This is essentially the same
      code borrowed from the watchman and eden getdeps.py.
      
      Reviewed By: simpkins
      
      Differential Revision: D14690997
      
      fbshipit-source-id: 3d3ae146237a2cd49609aaa2bd0e785ebe21f02c
      b499111d
    • Wez Furlong's avatar
      fbcode_builder: getdeps: add copytree helper · a520cc43
      Wez Furlong authored
      Summary:
      this module adds some functions that help with copying
      directory trees.  The copytree function is aware of eden and will
      issue a prefetch prior to walking the directory.
      
      Reviewed By: simpkins
      
      Differential Revision: D14691006
      
      fbshipit-source-id: 079bf850756f61aca17978453d07bc73b2f91788
      a520cc43
    • Wez Furlong's avatar
      fbcode_builder: getdeps: add run_cmd() function · dc89e337
      Wez Furlong authored
      Summary:
      This runs a command, raising an exception if it exits with a non-zero error status.
      
      It prints out the arguments in a mostly copy-and-pasteable form, with PATH-like
      env vars pretty printed to make it easier to see what is being invoked; here's
      an example of how cmake is being invoked later in this stack:
      
      ```
       ---
      + CMAKE_PREFIX_PATH=\
      +      /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/installed/ninja-5d7ec7:\
      +      /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/installed/cmake-91dc9a:\
      + PKG_CONFIG_PATH=\
      +      /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/installed/ninja-5d7ec7/lib/pkgconfig:\
      +      /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/installed/cmake-91dc9a/lib/pkgconfig:\
      + cd /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/build/zstd-470344 && \
      + cmake configure /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/repos/github.com-facebook-zstd.git/build/cmake -DCMAKE_INST
      ALL_PREFIX=/data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/installed/zstd-470344 -DBUILD_SHARED_LIBS=OFF -DCMAKE_BUILD_TYPE=R
      elWithDebInfo -G Ninja
      ```
      
      Reviewed By: simpkins
      
      Differential Revision: D14690999
      
      fbshipit-source-id: cdb0c681c7dfdfdc6e8c96bf4830bfbcf666411b
      dc89e337
    • Wez Furlong's avatar
      fbcode_builder: add getdeps main entrypoint · 7208ce46
      Wez Furlong authored
      Summary:
      This is the new getdeps entrypoint.  It lives in `opensource/fbcode_builder` so that
      it is synced into the various opensource projects at FB.
      
      In the incarnation in this diff is has a single subcommand that can be used to validate
      a manifest file.
      
      Reviewed By: sinancepel
      
      Differential Revision: D14690995
      
      fbshipit-source-id: 53f8ec9bafb7b1930970079e3ce20f496ac1c188
      7208ce46
    • Wez Furlong's avatar
      fbcode_builder: getdeps: add envfuncs.py · 496a8fa7
      Wez Furlong authored
      Summary:
      Add some helpers for manipulating environment variables
      that will hold either compiler flags or paths.
      
      These are used in later diffs when setting up arguments/environment
      for eg: cmake.
      
      Reviewed By: sinancepel, simpkins
      
      Differential Revision: D14690993
      
      fbshipit-source-id: 7f9753cd99d968550fe9e3ba8b7017a44683061e
      496a8fa7
    • Wez Furlong's avatar
      fbcode_builder: add manifest files for some dependencies · 003da387
      Wez Furlong authored
      Summary:
      These are ported over from the logic in the watchman and eden getdeps
      scripts, with additions to help bootstrap a build environment.
      
      These are sufficient to build watchman with thrift support on windows, mac and
      linux, and eden on mac and linux when combined with the getdeps code that
      follows in later diffs in this stack.
      
      Reviewed By: simpkins
      
      Differential Revision: D14691005
      
      fbshipit-source-id: 7f8b02fedcdc020e2d0e758c466959d8161d4587
      003da387
    • Wez Furlong's avatar
      fbcode_builder: add manifest file parser · e139393e
      Wez Furlong authored
      Summary:
      Adds a parser for manifest files that describe projects
      that may be either 1st party or 3rd party.
      
      A selection of manifest files appears in a later diff in this stack.
      
      This diff provides the raw parser and a couple of low level helpers.
      It also defines a mechanism for validating the schema to help catch
      structural (rather than semantic) issues with manifest file contents.
      
      Later diffs in the stack add helpers for accessing the data in
      a higher level way.
      
      Reviewed By: sinancepel
      
      Differential Revision: D14691003
      
      fbshipit-source-id: 7d2930a3359ede0f6e21fdc45686b083ab7a9ffa
      e139393e
    • Wez Furlong's avatar
      fbcode_builder: getdeps: add boolean expression parser · 5aab1c26
      Wez Furlong authored
      Summary:
      As part of folding getdeps into fbcode_builder, this
      expression parser is needed to allow constrained and deterministic
      conditionals in the manifest file format.
      
      For example, the watchman manifest will use this cargo-inspired syntax
      for system dependent sections:
      
      ```
      [dependencies]
      folly
      
      [dependencies.not(os=windows)]
      thrift
      ```
      
      Reviewed By: sinancepel
      
      Differential Revision: D14691014
      
      fbshipit-source-id: 080bcdb20579da40d225799f5f22debe65708b03
      5aab1c26
    • Chad Austin's avatar
      detect at runtime whether SKIP should fail · 43e2ad6e
      Chad Austin authored
      Summary:
      Detect at runtime whether testpilot is the test runner to decide how
      to communicate skip status.
      
      Reviewed By: simpkins
      
      Differential Revision: D15073229
      
      fbshipit-source-id: 417a1f4e42d0655e25cc62b0abb016c157d1adea
      43e2ad6e
    • Erik Hortsch's avatar
      Allow the tick interval to be specified on folly EventBase · c0b4901a
      Erik Hortsch authored
      Summary:
      Allow the eventbase to be constructed with a higher-precision timer, for those callers that need more precision for shorter (or higher entropy) waits.
      
      (Note: this ignores all push blocking failures!)
      
      Differential Revision: D15186374
      
      fbshipit-source-id: 98cca2d15fc18f9e865129f4629a4351e9a8b9c9
      c0b4901a
    • Kirk Shoop's avatar
      fixes for take_until tests · 578c4b91
      Kirk Shoop authored
      Summary:
      there were test failures that fell into three buckets
      
      first, the strands worker had concurrency issues. the handoff between submit and the worker was wrong.
      
      second, take_until was not waiting for the producer to signal done. this was originally by-design. now the expectation is that take until will not early-complete on cancellation.
      
      third, the tests had concurrency issues. the EXPECT assertions were not allowing for non-deterministic interleaving.
      
      Reviewed By: lewissbaker
      
      Differential Revision: D14947473
      
      fbshipit-source-id: 36f7cf87b9226367dc44afed663fa6418503c059
      578c4b91
  2. 02 May, 2019 4 commits
    • Yedidya Feldblum's avatar
      Let FBVector.h support -fno-exceptions · 1299eec8
      Yedidya Feldblum authored
      Summary: [Folly] Let `folly/FBVector.h` support `-fno-exceptions`. Same mechanism as with `folly/small_vector.h`.
      
      Reviewed By: Orvid
      
      Differential Revision: D15158754
      
      fbshipit-source-id: fe67d608a5b38258d05718efae502b7277e1f4ee
      1299eec8
    • Andrii Grynenko's avatar
      Add co_awaitTry support for folly::Future · be621139
      Andrii Grynenko authored
      Summary: This also makes co_awaitTry to work for any SemiAwaitable that implements await_resume_try in it's awaiter.
      
      Reviewed By: yfeldblum
      
      Differential Revision: D15170076
      
      fbshipit-source-id: 7f6ce0e2fa762acc5bcc4a32ed29d189787bf1ef
      be621139
    • Kirk Shoop's avatar
      improve pushmi compile-time debugging · ec7519ab
      Kirk Shoop authored
      Summary:
      use PUSHMI_TEMPLATE_DEBUG in a few more places
      add some additional code to the submit customization point to allow the compiler to emit error info when none of the fallbacks worked.
      
      Reviewed By: yfeldblum
      
      Differential Revision: D15016463
      
      fbshipit-source-id: 1c43c21093d54d662c1d084040608db0e0c029be
      ec7519ab
    • Junqi Wang's avatar
      Back out "[Folly] Enforce SingletonThreadLocal uniqueness without FOLLY_TLS" · c43e3f85
      Junqi Wang authored
      Summary: Original commit changeset: 04dbc7bc7752
      
      Reviewed By: vchynarov
      
      Differential Revision: D15168113
      
      fbshipit-source-id: a69618eabec30e7599c13bb32c25f6efd6da561e
      c43e3f85
  3. 01 May, 2019 3 commits
    • Lee Howes's avatar
      Check then callability using tryCallableResult as it forwards to thenTry anyway · 7406e351
      Lee Howes authored
      Summary: Switch to the correct callability check to match the internals. Without this change then is ambiguous and can still generate errors on polymorphic lambdas.
      
      Reviewed By: yfeldblum
      
      Differential Revision: D15147830
      
      fbshipit-source-id: 3f577641495a24076e9ffa19deaad190452c420c
      7406e351
    • Chad Austin's avatar
      allow conversion from nullptr to folly::Optional<bool> · 67210f1c
      Chad Austin authored
      Summary:
      Disabling the nullptr conversion to non-pointer folly::Optional<T> has
      baked long enough. Re-enabled it with the same behavior that
      std::optional has.
      
      Reviewed By: Orvid
      
      Differential Revision: D15151503
      
      fbshipit-source-id: d170f3f0e6bf3bb4461f17bdb6f8078b5a883615
      67210f1c
    • Dan Melnic's avatar
      Fix SingletonBenchmark.cpp init · 5f8a9671
      Dan Melnic authored
      Summary: Call folly::SingletonVault::singleton<BenchmarkTag>()->registrationComplete() to avoid an assert
      
      Reviewed By: yfeldblum
      
      Differential Revision: D15159381
      
      fbshipit-source-id: 6f3be22d433e77d6e1b5293098a6dedd42e65bb6
      5f8a9671
  4. 30 Apr, 2019 6 commits